Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/latest/pull_cdc_integration_pulsar_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-l4m71-6308n --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "65b81d230d83741f43dcfd42a992705f70aa8201" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-l4m71" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-list --no-walk b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} _=/usr/bin/printenv TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-l4m71 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=4ad1b1c6-1b6c-42ca-b920-80c586089d25 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Debug info JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=3 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-l4m71 pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-100f7bb7add7cfb003dff5768bf20348b91a6b6cef38154d1625c661ef800a11 + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.23.3' GODEBUG='' GOTELEMETRY='local' GOTELEMETRYDIR='/home/jenkins/.config/go/telemetry' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3286331447=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 bash' debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-l4m7-lh3l0 bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1138 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 9c6b4c0379426e4b (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 24 IN A 20.27.177.113 github.com. 24 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 04 07:24:02 UTC 2025 ;; MSG SIZE rcvd: 103 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12120/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [.github/workflows/check_and_build.yaml, .github/workflows/dataflow_engine_chaos.yaml, .github/workflows/dm_binlog_999999.yaml, .github/workflows/dm_chaos.yaml, .github/workflows/dm_mariadb_master_down_and_up.yaml, .github/workflows/dm_upstream_switch.yaml] [Pipeline] echo diff file not matched: .github/workflows/check_and_build.yaml [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-4c631d5-a306f48) 192337920 bytes in 0.71 secs (272650291 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.43.5 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/.git/ .git HEAD is now at a306f4894 dm: upgrade go-mysql to v1.12.0 POST git-upload-pack (656 bytes) POST git-upload-pack (973 bytes) From https://github.com/pingcap/tiflow = [up to date] master -> origin/master * [new ref] refs/pull/12120/head -> origin/pr/12120/head Previous HEAD position was a306f4894 dm: upgrade go-mysql to v1.12.0 HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) ๐Ÿšง Checkouting to base SHA:4c631d5951854ac91765b729fe3849d8e8d5ed8a... HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 4c631d5951854ac91765b729fe3849d8e8d5ed8a 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) 9ed3d76c5 consumer(ticdc): adjust consumer log level (#12092) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 4c631d5951854ac91765b729fe3849d8e8d5ed8a ... Updating 4c631d595..e8bfbe4ba Fast-forward .github/workflows/check_and_build.yaml | 2 +- .github/workflows/dataflow_engine_chaos.yaml | 2 +- .github/workflows/dm_binlog_999999.yaml | 4 ++-- .github/workflows/dm_chaos.yaml | 4 ++-- .github/workflows/dm_mariadb_master_down_and_up.yaml | 4 ++-- .github/workflows/dm_upstream_switch.yaml | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) ๐Ÿงพ Pre-merged result: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 e8bfbe4ba CI: Upgrade to actions/cache@v4 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache saved successfully (git/pingcap/tiflow/rev-4c631d5-e8bfbe4) 192358400 bytes in 2.84 secs (67705470 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/third_party_download [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] echo computeBranchFromPR component: tidb, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: pd, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tikv, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tiflash, prTargetBranch: master, prTitle: CI: Upgrade to actions/cache@v4, trunkBranch: master [Pipeline] retry [Pipeline] { [Pipeline] sh + export TIDB_BRANCH=master + TIDB_BRANCH=master + export PD_BRANCH=master + PD_BRANCH=master + export TIKV_BRANCH=master + TIKV_BRANCH=master + export TIFLASH_BRANCH=master + TIFLASH_BRANCH=master + cd ../tiflow + ./scripts/download-integration-test-binaries.sh master Downloading binaries... >>> Downloading tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz 2025-04-04 15:24:31 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz [620275338/620275338] -> "tmp/tidb-server.tar.gz" [1] >>> Downloading pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz 2025-04-04 15:24:47 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz [228444265/228444265] -> "tmp/pd-server.tar.gz" [1] >>> Downloading tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz 2025-04-04 15:25:09 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz [817400808/817400808] -> "tmp/tikv-server.tar.gz" [1] >>> Downloading tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz 2025-04-04 15:25:33 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz [553235170/553235170] -> "tmp/tiflash.tar.gz" [1] >>> Downloading minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2025-04-04 15:25:39 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> Downloading etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2025-04-04 15:25:40 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> Downloading schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2025-04-04 15:25:49 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] >>> Downloading go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> Downloading jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2025-04-04 15:25:52 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] Download SUCCESS + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + make check_third_party_binary go: downloading go1.23.4 (linux/amd64) /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-region-bench ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/pd-ut ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ../tiflow/bin/xprog ./bin/ + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:25 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:25 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + ./bin/tidb-server -V Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + ./bin/tikv-server -V TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:25 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:59" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/IBM/sarama v1.41.2 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/xdg/scram v1.0.5 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/google/btree v1.1.2 go: downloading cloud.google.com/go v0.112.2 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/docker/go-units v0.5.0 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading k8s.io/apimachinery v0.29.11 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_pulsar_consumer ']' + make pulsar_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:10" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer go: downloading github.com/confluentinc/confluent-kafka-go/v2 v2.3.0 go: downloading github.com/DataDog/zstd v1.5.5 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/sync_diff_inspector ./sync_diff_inspector go: downloading github.com/olekukonko/tablewriter v0.0.5 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20241224111350-ad2e4f976c7c go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading gorm.io/driver/mysql v1.5.7 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/libc v1.37.1 go: downloading modernc.org/sqlite v1.27.0 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:28:18" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e8bfbe4ba98e05140ab733ee3d08dd3b99918721" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ge8bfbe4ba"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.4G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:35 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:35 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:35 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:30 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:30 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:28 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:30 oauth2-server -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:30 sync_diff_inspector + ./bin/cdc version Release Version: v9.0.0-beta.1-10-ge8bfbe4ba Git Commit Hash: e8bfbe4ba98e05140ab733ee3d08dd3b99918721 Git Branch: HEAD UTC Build Time: 2025-04-04 07:28:18 Go Version: go1.23.4 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-pulsar-test/rev-4c631d5-e8bfbe4) 1485424640 bytes in 18.02 secs (82450868 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-region-bench ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/pd-ut ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ../third_party_download/bin/xprog ./bin/ + ls -alh ./bin total 3.3G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:35 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:25 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:35 bin -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:35 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:35 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:30 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:30 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:28 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:35 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 4 15:35 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:35 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:35 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:35 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:35 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Apr 4 15:35 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 4 15:35 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Apr 4 15:35 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:35 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 4 15:35 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 4 15:35 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 4 15:35 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 4 15:35 minio -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:30 oauth2-server -rwxr-xr-x 1 jenkins jenkins 41M Apr 4 15:35 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:35 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 4 15:35 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:35 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:35 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 4 15:35 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:35 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 4 15:35 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:35 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:35 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:35 stores-dump -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:30 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 241M Apr 4 15:35 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 4 15:35 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 4 15:35 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 4 15:35 xprog [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 49.12 secs (82062519 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-460x2-x6bjk --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ecfbce60067d74fc26639d0c0f71c800a38415a2" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-460x2" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test [Pipeline] { Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 [Pipeline] checkout The recommended git tool is: git Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-pktl7-53n6r --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ed943cd234935302236991660af4aa92b9dacbda" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-pktl7" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test [Pipeline] podTemplate [Pipeline] { [Pipeline] { [Pipeline] node [Pipeline] checkout The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@722aa740; decorates RemoteLauncher[hudson.remoting.Channel@4fa3f28f:JNLP4-connect connection from 10.233.86.102/10.233.86.102:52824] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7b94df7b; decorates RemoteLauncher[hudson.remoting.Channel@7f5431b1:JNLP4-connect connection from 10.233.67.254/10.233.67.254:38910] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 [Pipeline] podTemplate Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" [Pipeline] { [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] stage [Pipeline] { (Test) Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6mk3h-0n904 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e9bf43fdf6eb7852efac12141ac0f1821a79ddf4" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6mk3h" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-frs04-5c16j --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "a39fe330f1245ed89b08877f7e94c8d36c8fe33c" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-frs04" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k8vqd-3572f --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "cec407f859ee80cd346fd6ef8043dfef0c853a33" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k8vqd" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-kpwn8-xsqsf --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "0707b8afd78bee5b0f2bcabb0af89912d0b8d36a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-kpwn8" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k7htz-f5rrs --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e260cb6d5dc670fe9ca0382dce760c64cbc08606" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k7htz" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-n5036-fqrd5 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "b7d5d66bb27dd80c7029c15b08e55c2a5041f681" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-n5036" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-fl57n-fgkl6 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "bb3f9ad971322ee12d32184c5893631dbcf611b5" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-fl57n" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-mdsrk-k02rv --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c9e7e23d7223fcb2bc08065dbf532fa26e0f42db" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-mdsrk" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 6.72 secs (600128942 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G00 [Pipeline] node [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] container Run cases: down_db_require_secure_transport bdr_mode capture_suicide_while_balance_table syncpoint syncpoint_check_ts server_config_compatibility changefeed_dup_error_restart safe_mode kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update csv_storage_update_pk_clustered csv_storage_update_pk_nonclustered GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-460x2 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=5b190d6e-ec78-4b30-9587-cf3fc221f01a GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G00 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-460x-f7tcb pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-460x2 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/down_db_require_secure_transport/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:36:58 CST 2025] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] { [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5ac5b2f9; decorates RemoteLauncher[hudson.remoting.Channel@2080a7c6:JNLP4-connect connection from 10.233.78.16/10.233.78.16:35208] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1fd5e628; decorates RemoteLauncher[hudson.remoting.Channel@1011cdb1:JNLP4-connect connection from 10.233.106.237/10.233.106.237:56744] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@524e9761; decorates RemoteLauncher[hudson.remoting.Channel@3ee18998:JNLP4-connect connection from 10.233.105.51/10.233.105.51:38654] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3a4f66e; decorates RemoteLauncher[hudson.remoting.Channel@32cddcbc:JNLP4-connect connection from 10.233.84.25/10.233.84.25:33906] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@534ff997; decorates RemoteLauncher[hudson.remoting.Channel@4b3692e9:JNLP4-connect connection from 10.233.100.215/10.233.100.215:59728] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@54f90a05; decorates RemoteLauncher[hudson.remoting.Channel@8b97266:JNLP4-connect connection from 10.233.69.136/10.233.69.136:33786] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6db051a2; decorates RemoteLauncher[hudson.remoting.Channel@4be125d6:JNLP4-connect connection from 10.233.123.184/10.233.123.184:35818] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@74ecf28e; decorates RemoteLauncher[hudson.remoting.Channel@469de885:JNLP4-connect connection from 10.233.70.76/10.233.70.76:48150] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-s05zz-jg0jq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "90bf458e7c18cac08f279e75323b28ce9d1a1a73" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-s05zz" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-vsw6p-wmqjs --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "fdace1b16e2af3edcab9a48e8a5ac1efbb97fd99" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-vsw6p" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k6tq6-fcsck --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f5a971202d9dca912bd440dc674cfa8efcf51fbc" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k6tq6" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:02 CST 2025] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-wb2gs-lg28n --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "86c4387546ddd8eb24af344b1072dd85aa7dce43" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-wb2gs" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-cf30g-8xm94 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ce036475304e466d47d5ae9d983fa4ee7222e09f" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-cf30g" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-xz56z-wbxn8 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "5290353584dbff8ad529868d4047a6d776134f1e" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-xz56z" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6cb1w-0kxzc --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e22e20df5d914728d3fca6e71096eaf9cfd6b4ea" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6cb1w" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 is provisioned from template pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-bmrsl-xmfdc --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3aabec86d43157b2808c984f710015d418ed0cf0" jenkins/label: "pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-bmrsl" name: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "12" requests: memory: "32Gi" cpu: "12" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test Running on pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: pulsar... <<================= kafka downstream isn't support syncpoint record [Fri Apr 4 15:37:05 CST 2025] <<<<<< run test case syncpoint success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/syncpoint_check_ts/run.sh using Sink-Type: pulsar... <<================= kafka downstream isn't support syncpoint record [Fri Apr 4 15:37:09 CST 2025] <<<<<< run test case syncpoint_check_ts success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.91 secs (509482677 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G01 [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth debezium_basic kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-pktl7 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=aa534f8d-fa16-4c0c-8471-a053103a30be GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G01 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-pktl7 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:10 CST 2025] <<<<<< run test case http_api success! >>>>>> [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7415e1d1; decorates RemoteLauncher[hudson.remoting.Channel@1e9de5bb:JNLP4-connect connection from 10.233.126.124/10.233.126.124:41364] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4e728b27; decorates RemoteLauncher[hudson.remoting.Channel@6993565:JNLP4-connect connection from 10.233.93.104/10.233.93.104:41860] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@cab06d2; decorates RemoteLauncher[hudson.remoting.Channel@592d867f:JNLP4-connect connection from 10.233.108.52/10.233.108.52:58672] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7c33ad80; decorates RemoteLauncher[hudson.remoting.Channel@44876b01:JNLP4-connect connection from 10.233.76.117/10.233.76.117:60884] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@41b97852; decorates RemoteLauncher[hudson.remoting.Channel@15093e8c:JNLP4-connect connection from 10.233.108.37/10.233.108.37:36736] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@43c8d5e5; decorates RemoteLauncher[hudson.remoting.Channel@14bf6662:JNLP4-connect connection from 10.233.71.51/10.233.71.51:48732] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@68a85227; decorates RemoteLauncher[hudson.remoting.Channel@6f0c3a6f:JNLP4-connect connection from 10.233.68.6/10.233.68.6:33940] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4fd6d212; decorates RemoteLauncher[hudson.remoting.Channel@13edfb5f:JNLP4-connect connection from 10.233.72.183/10.233.72.183:43752] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:12 CST 2025] <<<<<< run test case server_config_compatibility success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:13 CST 2025] <<<<<< run test case http_api_tls success! >>>>>> Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:16 CST 2025] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/safe_mode/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:20 CST 2025] <<<<<< run test case safe_mode success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:20 CST 2025] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.26 secs (487694192 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:23 CST 2025] <<<<<< run test case kafka_big_messages success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:27 CST 2025] <<<<<< run test case kafka_compression success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/debezium_basic/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:27 CST 2025] <<<<<< run test case debezium_basic success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:30 CST 2025] <<<<<< run test case kafka_simple_basic success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:30 CST 2025] <<<<<< run test case kafka_messages success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 9.15 secs (440259361 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:33 CST 2025] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:34 CST 2025] <<<<<< run test case kafka_sink_error_resume success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:37 CST 2025] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:37 CST 2025] <<<<<< run test case mq_sink_lost_callback success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:40 CST 2025] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:41 CST 2025] <<<<<< run test case mq_sink_dispatcher success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 6.80 secs (592907719 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:43 CST 2025] <<<<<< run test case kafka_simple_claim_check success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:45 CST 2025] <<<<<< run test case kafka_column_selector success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:47 CST 2025] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:48 CST 2025] <<<<<< run test case kafka_column_selector_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:50 CST 2025] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.38 secs (546084928 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:52 CST 2025] <<<<<< run test case lossy_ddl success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:55 CST 2025] <<<<<< run test case storage_csv_update success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/canal_json_basic Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:37:59 CST 2025] <<<<<< run test case csv_storage_update_pk_clustered success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.30 secs (485505356 bytes/sec) [Pipeline] { [Pipeline] cache Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:38:02 CST 2025] <<<<<< run test case csv_storage_update_pk_nonclustered success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359009135991792 TiDB Cluster ID. tikv_gc_leader_uuid 657fef12a3c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:2583, start at 2025-04-04 15:38:04.845001422 +0800 CST m=+1.714895738 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:04.853 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:04.815 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:04.815 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359009135991792 TiDB Cluster ID. tikv_gc_leader_uuid 657fef12a3c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:2583, start at 2025-04-04 15:38:04.845001422 +0800 CST m=+1.714895738 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:04.853 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:04.815 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:04.815 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359015904121202 TiDB Cluster ID. tikv_gc_leader_uuid 657fef12bc80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:2645, start at 2025-04-04 15:38:04.922116558 +0800 CST m=+1.727821648 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:04.931 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:04.914 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:04.914 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Fri Apr 4 15:38:07 CST 2025] <<<<<< START cdc server in canal_json_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.39383940.out server --log-file /tmp/tidb_cdc_test/canal_json_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_basic/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:10 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f025f1-d862-4559-8ce1-9e9be0d4f833 {"id":"47f025f1-d862-4559-8ce1-9e9be0d4f833","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752287} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc2fc6cd 47f025f1-d862-4559-8ce1-9e9be0d4f833 /tidb/cdc/default/default/upstream/7489359009135991792 {"id":7489359009135991792,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f025f1-d862-4559-8ce1-9e9be0d4f833 {"id":"47f025f1-d862-4559-8ce1-9e9be0d4f833","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752287} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc2fc6cd 47f025f1-d862-4559-8ce1-9e9be0d4f833 /tidb/cdc/default/default/upstream/7489359009135991792 {"id":7489359009135991792,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47f025f1-d862-4559-8ce1-9e9be0d4f833 {"id":"47f025f1-d862-4559-8ce1-9e9be0d4f833","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752287} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc2fc6cd 47f025f1-d862-4559-8ce1-9e9be0d4f833 /tidb/cdc/default/default/upstream/7489359009135991792 {"id":7489359009135991792,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:10 CST 2025] <<<<<< START pulsar cluster in canal_json_basic case >>>>>>' [Fri Apr 4 15:38:10 CST 2025] <<<<<< START pulsar cluster in canal_json_basic case >>>>>> + workdir=/tmp/tidb_cdc_test/canal_json_basic + cluster_type=normal + cd /tmp/tidb_cdc_test/canal_json_basic + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/canal_json_basic/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/canal_json_basic/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/canal_json_basic/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/canal_json_basic/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/canal_json_basic/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/canal_json_basic/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/canal_json_basic/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/canal_json_basic/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/canal_json_basic/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:10 CST 2025] <<<<<< START pulsar cluster in normal mode in canal_json_basic case >>>>>>' [Fri Apr 4 15:38:10 CST 2025] <<<<<< START pulsar cluster in normal mode in canal_json_basic case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/canal_json_basic/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/canal_json_basic/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/canal_json_basic/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.14 secs (564241250 bytes/sec) [Pipeline] { [Pipeline] cache + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.04 secs (572676223 bytes/sec) [Pipeline] { [Pipeline] cache public/default ++ date + echo '[Fri Apr 4 15:38:21 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:21 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.cli.4515.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-canal-json-basic?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 49fb6861-b648-4ff7-9f27-811d13ade0d2 Info: {"upstream_id":7489359009135991792,"namespace":"default","id":"49fb6861-b648-4ff7-9f27-811d13ade0d2","sink_uri":"pulsar://127.0.0.1:6650/ticdc-canal-json-basic?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:38:22.18536284+08:00","start_ts":457114203354824706,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114203354824706,"checkpoint_ts":457114203354824706,"checkpoint_time":"2025-04-04 15:38:21.616"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x [Fri Apr 4 15:38:28 CST 2025] <<<<<< START Pulsar consumer in canal_json_basic case >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 9.17 secs (439736122 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G02 [Pipeline] sh Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6mk3h HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=faa60720-6209-4e1d-9443-280ab2e10285 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G02 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6mk3h pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6mk3-f3330 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:38:32 CST 2025] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G05 [Pipeline] sh Run cases: charset_gbk ddl_manager multi_source vector GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-frs04 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=96754cac-f76f-4354-a4fd-5e85ae39f668 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G05 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-frs04 pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G03 [Pipeline] sh Run cases: row_format drop_many_tables processor_stop_delay partition_table ddl_with_exists GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k8vqd HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=45d7e3ca-e519-4694-a12e-0c760f55c114 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G03 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k8vqd PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G04 [Pipeline] sh Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-kpwn8 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=ef42eecc-64d2-4692-8ba2-dc6dfb9bf2a9 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G04 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-kpwn8 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G06 [Pipeline] sh Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k7htz HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=917c60e8-8a06-4e31-9ed0-aa386dbd39d3 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G06 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k7htz PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G07 [Pipeline] sh Run cases: kv_client_stream_reconnect cdc split_region GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-n5036 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=4447a6c4-745f-4978-9b31-3f7c9aafbcfe GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G07 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-n5036 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-fl57n HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=96688cc6-1fd8-4e1b-b313-fd1b7026f3d6 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G08 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-fl57n PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G09 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-mdsrk HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=cd5974a5-5b7d-44e1-a7c6-8e2e2856a733 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G09 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-mdsrk PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] // cache [Pipeline] } [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: pulsar... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] // dir [Pipeline] } [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330D7EA436A6AA < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:38:38 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Bucket 's3://logbucket/' created [Fri Apr 4 15:38:38 CST 2025] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: pulsar... <<================= Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Fri Apr 4 15:38:41 CST 2025] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359185143496537 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39188000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:1429, start at 2025-04-04 15:38:44.234278814 +0800 CST m=+1.710315488 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.245 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.243 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.243 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359185143496537 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39188000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:1429, start at 2025-04-04 15:38:44.234278814 +0800 CST m=+1.710315488 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.245 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.243 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.243 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359178623942791 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3930c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:1497, start at 2025-04-04 15:38:44.321961862 +0800 CST m=+1.730161130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.331 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.291 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.291 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359185106667618 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39640000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:1405, start at 2025-04-04 15:38:44.512221665 +0800 CST m=+1.587060606 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.519 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186010764108 TiDB Cluster ID. tikv_gc_leader_uuid 657fef390f00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:1487, start at 2025-04-04 15:38:44.163801228 +0800 CST m=+1.539162693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.172 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.156 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.156 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186010764108 TiDB Cluster ID. tikv_gc_leader_uuid 657fef390f00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:1487, start at 2025-04-04 15:38:44.163801228 +0800 CST m=+1.539162693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.172 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.156 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.156 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359189063935823 TiDB Cluster ID. tikv_gc_leader_uuid 657fef391b00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:1564, start at 2025-04-04 15:38:44.209875369 +0800 CST m=+1.509770043 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.204 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.204 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359185106667618 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39640000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:1405, start at 2025-04-04 15:38:44.512221665 +0800 CST m=+1.587060606 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.519 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359178004984260 TiDB Cluster ID. tikv_gc_leader_uuid 657fef398c40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:1477, start at 2025-04-04 15:38:44.669281069 +0800 CST m=+1.666956677 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.678 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.657 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.657 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:38:45 CST 2025] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186351336507 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39e4c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:1424, start at 2025-04-04 15:38:45.032936312 +0800 CST m=+1.597218728 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.011 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.011 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186351336507 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39e4c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:1424, start at 2025-04-04 15:38:45.032936312 +0800 CST m=+1.597218728 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.011 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.011 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359190210865239 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39f2c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:1502, start at 2025-04-04 15:38:45.084178111 +0800 CST m=+1.578757575 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.093 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.067 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.067 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359187832153176 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3a1780007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:1355, start at 2025-04-04 15:38:45.226674875 +0800 CST m=+1.629605091 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.234 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.214 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.214 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359187832153176 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3a1780007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:1355, start at 2025-04-04 15:38:45.226674875 +0800 CST m=+1.629605091 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.234 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.214 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.214 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359183796924894 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3a160000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:1419, start at 2025-04-04 15:38:45.238067827 +0800 CST m=+1.558578720 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.246 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186598347952 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39bdc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:1501, start at 2025-04-04 15:38:44.863500318 +0800 CST m=+1.628330815 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.872 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.855 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.855 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359186598347952 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39bdc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:1501, start at 2025-04-04 15:38:44.863500318 +0800 CST m=+1.628330815 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.872 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.855 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.855 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359188376185627 TiDB Cluster ID. tikv_gc_leader_uuid 657fef39d6c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:1567, start at 2025-04-04 15:38:44.962412589 +0800 CST m=+1.657552910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:44.972 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:44.955 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:44.955 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 2-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359184551411510 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3a7c80005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:1499, start at 2025-04-04 15:38:45.625398929 +0800 CST m=+1.581536810 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.633 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359184551411510 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3a7c80005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:1499, start at 2025-04-04 15:38:45.625398929 +0800 CST m=+1.581536810 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.633 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359190965723685 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3aa24000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:1562, start at 2025-04-04 15:38:45.809965357 +0800 CST m=+1.695343764 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:45.818 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:45.820 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:45.820 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2858.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2884.out cli tso query --pd=http://127.0.0.1:2379 ++ date + echo '[Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in processor_err_chan case >>>>>>' [Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in processor_err_chan case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_err_chan + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_err_chan + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_err_chan/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_err_chan/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_err_chan/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_err_chan/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_err_chan/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_err_chan/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_err_chan/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_err_chan/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_err_chan/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_err_chan case >>>>>>' [Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_err_chan case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_err_chan/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_err_chan/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_err_chan/pulsar-bookie table test.finish_mark not exists for 3-th check, retry later + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Fri Apr 4 15:38:47 CST 2025] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.28652867.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in gc_safepoint case >>>>>>' [Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in gc_safepoint case >>>>>> + workdir=/tmp/tidb_cdc_test/gc_safepoint + cluster_type=normal + cd /tmp/tidb_cdc_test/gc_safepoint + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/gc_safepoint/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/gc_safepoint/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/gc_safepoint/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/gc_safepoint/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/gc_safepoint/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/gc_safepoint/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/gc_safepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/gc_safepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/gc_safepoint/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in normal mode in gc_safepoint case >>>>>>' [Fri Apr 4 15:38:47 CST 2025] <<<<<< START pulsar cluster in normal mode in gc_safepoint case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/gc_safepoint/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/gc_safepoint/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/gc_safepoint/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2868.out cli tso query --pd=http://127.0.0.1:2379 ++ date + echo '[Fri Apr 4 15:38:48 CST 2025] <<<<<< START pulsar cluster in kv_client_stream_reconnect case >>>>>>' [Fri Apr 4 15:38:48 CST 2025] <<<<<< START pulsar cluster in kv_client_stream_reconnect case >>>>>> + workdir=/tmp/tidb_cdc_test/kv_client_stream_reconnect + cluster_type=oauth + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar_standalone.conf + pulsar_port=6650 + '[' oauth == mtls ']' + '[' oauth == oauth ']' + echo 'Waiting for oauth2 server to be ready...' Waiting for oauth2 server to be ready... + i=0 + nc -z localhost 9096 + oauth2-server + gen_oauth_config + echo ' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ date + echo '[Fri Apr 4 15:38:48 CST 2025] <<<<<< START pulsar cluster in oauth mode in kv_client_stream_reconnect case >>>>>>' [Fri Apr 4 15:38:48 CST 2025] <<<<<< START pulsar cluster in oauth mode in kv_client_stream_reconnect case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.65 secs (465998533 bytes/sec) [Pipeline] { [Pipeline] cache + set +x + tso='457114210073051140 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114210073051140 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:38:48 CST 2025] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.29282930.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114210102411265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114210102411265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "mysql.port"="4000" "scanproportion"="0" "dotransactions"="false" "updateproportion"="0" "operationcount"="0" "workload"="core" "recordcount"="10" "readproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.db"="sink_retry" "threadcount"="2" "requestdistribution"="uniform" "insertproportion"="0" "mysql.user"="root" ********************************************** Run finished, takes 8.732437ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1797.2, Avg(us): 1622, Min(us): 944, Max(us): 4143, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:38:48 CST 2025] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.29172919.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:38:48 CST 2025] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 4-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + set +x + tso='457114210380021766 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114210380021766 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:38:49 CST 2025] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.29072909.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:50 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 {"id":"d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752327} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc77fcb d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 /tidb/cdc/default/default/upstream/7489359185106667618 {"id":7489359185106667618,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 {"id":"d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752327} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc77fcb d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 /tidb/cdc/default/default/upstream/7489359185106667618 {"id":7489359185106667618,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 {"id":"d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752327} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc77fcb d31c77f1-411a-4fa9-9a2e-5e8a8fe0feb0 /tidb/cdc/default/default/upstream/7489359185106667618 {"id":7489359185106667618,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:50 CST 2025] <<<<<< START pulsar cluster in charset_gbk case >>>>>>' [Fri Apr 4 15:38:50 CST 2025] <<<<<< START pulsar cluster in charset_gbk case >>>>>> + workdir=/tmp/tidb_cdc_test/charset_gbk + cluster_type=mtls + cd /tmp/tidb_cdc_test/charset_gbk + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/charset_gbk/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/charset_gbk/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/charset_gbk/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/charset_gbk/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/charset_gbk/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) .....................................................................+++++ .......................................................................................................+++++ e is 65537 (0x010001) + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ...............+++++ .......................................................+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=CN = server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ............................................................................................+++++ + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ...................................................................................................................................................................................+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=CN = broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/charset_gbk/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/charset_gbk/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem' ++ date + echo '[Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in mtls mode in charset_gbk case >>>>>>' [Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in mtls mode in charset_gbk case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/charset_gbk/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/charset_gbk/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/charset_gbk/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table test.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:51 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad3b78df-ec35-42e7-bece-d0edd72ee006 {"id":"ad3b78df-ec35-42e7-bece-d0edd72ee006","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc9c8d5 ad3b78df-ec35-42e7-bece-d0edd72ee006 /tidb/cdc/default/default/upstream/7489359185143496537 {"id":7489359185143496537,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad3b78df-ec35-42e7-bece-d0edd72ee006 {"id":"ad3b78df-ec35-42e7-bece-d0edd72ee006","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc9c8d5 ad3b78df-ec35-42e7-bece-d0edd72ee006 /tidb/cdc/default/default/upstream/7489359185143496537 {"id":7489359185143496537,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad3b78df-ec35-42e7-bece-d0edd72ee006 {"id":"ad3b78df-ec35-42e7-bece-d0edd72ee006","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcc9c8d5 ad3b78df-ec35-42e7-bece-d0edd72ee006 /tidb/cdc/default/default/upstream/7489359185143496537 {"id":7489359185143496537,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in foreign_key case >>>>>>' [Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in foreign_key case >>>>>> + workdir=/tmp/tidb_cdc_test/foreign_key + cluster_type=normal + cd /tmp/tidb_cdc_test/foreign_key + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/foreign_key/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/foreign_key/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/foreign_key/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/foreign_key/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/foreign_key/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/foreign_key/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/foreign_key/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/foreign_key/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/foreign_key/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in normal mode in foreign_key case >>>>>>' [Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in normal mode in foreign_key case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/foreign_key/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/foreign_key/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/foreign_key/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: pulsar... <<================= + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:51 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a701c0f6-523e-4e72-bac7-15e575f728f4 {"id":"a701c0f6-523e-4e72-bac7-15e575f728f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcca3be4 a701c0f6-523e-4e72-bac7-15e575f728f4 /tidb/cdc/default/default/upstream/7489359186010764108 {"id":7489359186010764108,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a701c0f6-523e-4e72-bac7-15e575f728f4 {"id":"a701c0f6-523e-4e72-bac7-15e575f728f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcca3be4 a701c0f6-523e-4e72-bac7-15e575f728f4 /tidb/cdc/default/default/upstream/7489359186010764108 {"id":7489359186010764108,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a701c0f6-523e-4e72-bac7-15e575f728f4 {"id":"a701c0f6-523e-4e72-bac7-15e575f728f4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752329} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcca3be4 a701c0f6-523e-4e72-bac7-15e575f728f4 /tidb/cdc/default/default/upstream/7489359186010764108 {"id":7489359186010764108,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in sink_retry case >>>>>>' [Fri Apr 4 15:38:51 CST 2025] <<<<<< START pulsar cluster in sink_retry case >>>>>> + workdir=/tmp/tidb_cdc_test/sink_retry + cluster_type=normal + cd /tmp/tidb_cdc_test/sink_retry + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/sink_retry/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/sink_retry/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/sink_retry/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/sink_retry/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/sink_retry/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/sink_retry/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/sink_retry/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/sink_retry/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/sink_retry/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:52 CST 2025] <<<<<< START pulsar cluster in normal mode in sink_retry case >>>>>>' [Fri Apr 4 15:38:52 CST 2025] <<<<<< START pulsar cluster in normal mode in sink_retry case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/sink_retry/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/sink_retry/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/sink_retry/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:52 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8ab2c8b1-b316-494b-84e0-b271cd4884cf {"id":"8ab2c8b1-b316-494b-84e0-b271cd4884cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752330} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc05d6 8ab2c8b1-b316-494b-84e0-b271cd4884cf /tidb/cdc/default/default/upstream/7489359187832153176 {"id":7489359187832153176,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8ab2c8b1-b316-494b-84e0-b271cd4884cf {"id":"8ab2c8b1-b316-494b-84e0-b271cd4884cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752330} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc05d6 8ab2c8b1-b316-494b-84e0-b271cd4884cf /tidb/cdc/default/default/upstream/7489359187832153176 {"id":7489359187832153176,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8ab2c8b1-b316-494b-84e0-b271cd4884cf {"id":"8ab2c8b1-b316-494b-84e0-b271cd4884cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752330} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc05d6 8ab2c8b1-b316-494b-84e0-b271cd4884cf /tidb/cdc/default/default/upstream/7489359187832153176 {"id":7489359187832153176,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:52 CST 2025] <<<<<< START pulsar cluster in row_format case >>>>>>' [Fri Apr 4 15:38:52 CST 2025] <<<<<< START pulsar cluster in row_format case >>>>>> + workdir=/tmp/tidb_cdc_test/row_format + cluster_type=normal + cd /tmp/tidb_cdc_test/row_format + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/row_format/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/row_format/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/row_format/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/row_format/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/row_format/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/row_format/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/row_format/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/row_format/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/row_format/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:38:53 CST 2025] <<<<<< START pulsar cluster in normal mode in row_format case >>>>>>' [Fri Apr 4 15:38:53 CST 2025] <<<<<< START pulsar cluster in normal mode in row_format case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/row_format/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/row_format/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/row_format/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + nc -z localhost 6651 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 public/default table test.finish_mark not exists for 6-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 public/default ++ date + echo '[Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>> * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330D8250669C11 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:38:54 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 4 15:38:54 CST 2025] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT ++ date + echo '[Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:38:54 CST 2025] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.32683270.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 public/default + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ++ date + echo '[Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:54 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:38:54 CST 2025] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.32843286.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 7-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.02 secs (574162765 bytes/sec) [Pipeline] { [Pipeline] cache + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public public/default =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:38:57 CST 2025] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:57 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3511044d-dd17-4136-9a0e-678952f18893 {"id":"3511044d-dd17-4136-9a0e-678952f18893","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752334} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcccfcf8 3511044d-dd17-4136-9a0e-678952f18893 /tidb/cdc/default/default/upstream/7489359186351336507 {"id":7489359186351336507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3511044d-dd17-4136-9a0e-678952f18893 {"id":"3511044d-dd17-4136-9a0e-678952f18893","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752334} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcccfcf8 3511044d-dd17-4136-9a0e-678952f18893 /tidb/cdc/default/default/upstream/7489359186351336507 {"id":7489359186351336507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3511044d-dd17-4136-9a0e-678952f18893 {"id":"3511044d-dd17-4136-9a0e-678952f18893","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752334} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcccfcf8 3511044d-dd17-4136-9a0e-678952f18893 /tidb/cdc/default/default/upstream/7489359186351336507 {"id":7489359186351336507,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:38:57 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:57 CST 2025] <<<<<< pulsar is ready >>>>>> table test.finish_mark not exists for 8-th check, retry later public/default Create changefeed successfully! ID: 122c9835-1373-4899-936c-49e3ad1e588c Info: {"upstream_id":7489359185106667618,"namespace":"default","id":"122c9835-1373-4899-936c-49e3ad1e588c","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:38:57.942612953+08:00","start_ts":457114210112897025,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114210112897025,"checkpoint_ts":457114210112897025,"checkpoint_time":"2025-04-04 15:38:47.396"} [Fri Apr 4 15:38:58 CST 2025] <<<<<< START Pulsar consumer in charset_gbk case >>>>>> Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:57 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/555725cf-fffc-4773-9bbd-49a9c327b4b8 {"id":"555725cf-fffc-4773-9bbd-49a9c327b4b8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccba5f7 555725cf-fffc-4773-9bbd-49a9c327b4b8 /tidb/cdc/default/default/upstream/7489359184551411510 {"id":7489359184551411510,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/555725cf-fffc-4773-9bbd-49a9c327b4b8 {"id":"555725cf-fffc-4773-9bbd-49a9c327b4b8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccba5f7 555725cf-fffc-4773-9bbd-49a9c327b4b8 /tidb/cdc/default/default/upstream/7489359184551411510 {"id":7489359184551411510,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/555725cf-fffc-4773-9bbd-49a9c327b4b8 {"id":"555725cf-fffc-4773-9bbd-49a9c327b4b8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccba5f7 555725cf-fffc-4773-9bbd-49a9c327b4b8 /tidb/cdc/default/default/upstream/7489359184551411510 {"id":7489359184551411510,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:38:58 CST 2025] <<<<<< START Pulsar consumer in gc_safepoint case >>>>>> 0 ++ date + echo '[Fri Apr 4 15:38:58 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:58 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.3376.out cli changefeed create --start-ts=457114210102411265 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-2285?protocol=canal-json&enable-tidb-extension=true' [Fri Apr 4 15:38:58 CST 2025] <<<<<< START Pulsar consumer in kv_client_stream_reconnect case >>>>>> check diff failed 1-th time, retry later Create changefeed successfully! ID: 413e3c16-9d93-400f-ad90-25836541ebf6 Info: {"upstream_id":7489359186010764108,"namespace":"default","id":"413e3c16-9d93-400f-ad90-25836541ebf6","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-2285?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:38:59.134045193+08:00","start_ts":457114210102411265,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114210102411265,"checkpoint_ts":457114210102411265,"checkpoint_time":"2025-04-04 15:38:47.356"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ++ date + echo '[Fri Apr 4 15:38:59 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:38:59 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.3388.out cli changefeed create --start-ts=457114210380021766 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-row-format-test-18123?protocol=canal-json&enable-tidb-extension=true' table test.finish_mark not exists for 9-th check, retry later Create changefeed successfully! ID: f60bc8c2-0639-4848-a49c-5db406a943b2 Info: {"upstream_id":7489359187832153176,"namespace":"default","id":"f60bc8c2-0639-4848-a49c-5db406a943b2","sink_uri":"pulsar://127.0.0.1:6650/ticdc-row-format-test-18123?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:39:00.293734987+08:00","start_ts":457114210380021766,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114210380021766,"checkpoint_ts":457114210380021766,"checkpoint_time":"2025-04-04 15:38:48.415"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:39:00 CST 2025] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.33953397.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x [Fri Apr 4 15:39:00 CST 2025] <<<<<< START Pulsar consumer in sink_retry case >>>>>> + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 10-th check, retry later + set +x check diff failed 2-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:01 CST 2025] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> [Fri Apr 4 15:39:01 CST 2025] <<<<<< START Pulsar consumer in row_format case >>>>>> public/default ++ date + echo '[Fri Apr 4 15:39:02 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:39:02 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.3486.out cli changefeed create --start-ts=457114210073051140 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-foreign-key-test-22803?protocol=canal-json&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/foreign_key/conf/cf.toml table test.finish_mark not exists for 11-th check, retry later Create changefeed successfully! ID: ed4c8cd6-7eea-4ab5-af56-b294d4279196 Info: {"upstream_id":7489359185143496537,"namespace":"default","id":"ed4c8cd6-7eea-4ab5-af56-b294d4279196","sink_uri":"pulsar://127.0.0.1:6650/ticdc-foreign-key-test-22803?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:39:03.526593377+08:00","start_ts":457114210073051140,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!foreign_key.t12"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114210073051140,"checkpoint_ts":457114210073051140,"checkpoint_time":"2025-04-04 15:38:47.244"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff successfully check_safepoint_forward http://127.0.0.1:2379 7489359186351336507 457114214062358533 457114212738531333 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:03 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8991d2d4-d2d2-4d23-bf72-54b58c200ea4 {"id":"8991d2d4-d2d2-4d23-bf72-54b58c200ea4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752340} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc6277 8991d2d4-d2d2-4d23-bf72-54b58c200ea4 /tidb/cdc/default/default/upstream/7489359186598347952 {"id":7489359186598347952,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8991d2d4-d2d2-4d23-bf72-54b58c200ea4 {"id":"8991d2d4-d2d2-4d23-bf72-54b58c200ea4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752340} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc6277 8991d2d4-d2d2-4d23-bf72-54b58c200ea4 /tidb/cdc/default/default/upstream/7489359186598347952 {"id":7489359186598347952,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8991d2d4-d2d2-4d23-bf72-54b58c200ea4 {"id":"8991d2d4-d2d2-4d23-bf72-54b58c200ea4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752340} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbccc6277 8991d2d4-d2d2-4d23-bf72-54b58c200ea4 /tidb/cdc/default/default/upstream/7489359186598347952 {"id":7489359186598347952,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:04 CST 2025] <<<<<< START Pulsar consumer in processor_err_chan case >>>>>> check_changefeed_state http://127.0.0.1:2379 b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea -s + info='{ "upstream_id": 7489359186598347952, "namespace": "default", "id": "b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea", "state": "normal", "checkpoint_tso": 457114214375096324, "checkpoint_time": "2025-04-04 15:39:03.655", "error": null }' + echo '{ "upstream_id": 7489359186598347952, "namespace": "default", "id": "b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea", "state": "normal", "checkpoint_tso": 457114214375096324, "checkpoint_time": "2025-04-04 15:39:03.655", "error": null }' { "upstream_id": 7489359186598347952, "namespace": "default", "id": "b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea", "state": "normal", "checkpoint_tso": 457114214375096324, "checkpoint_time": "2025-04-04 15:39:03.655", "error": null } ++ echo '{' '"upstream_id":' 7489359186598347952, '"namespace":' '"default",' '"id":' '"b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea",' '"state":' '"normal",' '"checkpoint_tso":' 457114214375096324, '"checkpoint_time":' '"2025-04-04' '15:39:03.655",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359186598347952, '"namespace":' '"default",' '"id":' '"b2d3d40e-42f3-4ecf-bf62-b5d8d5c3f6ea",' '"state":' '"normal",' '"checkpoint_tso":' 457114214375096324, '"checkpoint_time":' '"2025-04-04' '15:39:03.655",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later run task successfully [Fri Apr 4 15:39:04 CST 2025] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> check_changefeed_state http://127.0.0.1:2379 318da0c1-e8c8-467d-bb04-1e55e00d5de8 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=318da0c1-e8c8-467d-bb04-1e55e00d5de8 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 318da0c1-e8c8-467d-bb04-1e55e00d5de8 -s + set +x [Fri Apr 4 15:39:04 CST 2025] <<<<<< START Pulsar consumer in foreign_key case >>>>>> + info='{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' + echo '{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' { "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null } ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"stopped",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"stopped",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359186351336507 check diff failed 1-th time, retry later table test.finish_mark exists check diff successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 6.92 secs (582848092 bytes/sec) [Pipeline] { [Pipeline] cache table charset_gbk_test1.t0 not exists for 2-th check, retry later check diff failed 2-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:07 CST 2025] <<<<<< run test case multi_topics_v2 success! >>>>>> table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later run task successfully check_changefeed_state http://127.0.0.1:2379 318da0c1-e8c8-467d-bb04-1e55e00d5de8 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=318da0c1-e8c8-467d-bb04-1e55e00d5de8 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 318da0c1-e8c8-467d-bb04-1e55e00d5de8 -s + info='{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "normal", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' + echo '{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "normal", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' { "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "normal", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null } ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"normal",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"normal",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7489359186351336507 457114214062358533 457114214062358534 check diff failed 3-th time, retry later run task successfully check_changefeed_state http://127.0.0.1:2379 318da0c1-e8c8-467d-bb04-1e55e00d5de8 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=318da0c1-e8c8-467d-bb04-1e55e00d5de8 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 318da0c1-e8c8-467d-bb04-1e55e00d5de8 -s + info='{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' + echo '{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null }' { "upstream_id": 7489359186351336507, "namespace": "default", "id": "318da0c1-e8c8-467d-bb04-1e55e00d5de8", "state": "stopped", "checkpoint_tso": 457114214062358534, "checkpoint_time": "2025-04-04 15:39:02.462", "error": null } ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"stopped",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"318da0c1-e8c8-467d-bb04-1e55e00d5de8",' '"state":' '"stopped",' '"checkpoint_tso":' 457114214062358534, '"checkpoint_time":' '"2025-04-04' '15:39:02.462",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table test.finish_mark not exists for 2-th check, retry later check_changefeed_state http://127.0.0.1:2379 54a10e88-74dd-4044-985d-3140bcb34de2 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=54a10e88-74dd-4044-985d-3140bcb34de2 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 54a10e88-74dd-4044-985d-3140bcb34de2 -s + info='{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "54a10e88-74dd-4044-985d-3140bcb34de2", "state": "normal", "checkpoint_tso": 457114216028176388, "checkpoint_time": "2025-04-04 15:39:09.961", "error": null }' + echo '{ "upstream_id": 7489359186351336507, "namespace": "default", "id": "54a10e88-74dd-4044-985d-3140bcb34de2", "state": "normal", "checkpoint_tso": 457114216028176388, "checkpoint_time": "2025-04-04 15:39:09.961", "error": null }' { "upstream_id": 7489359186351336507, "namespace": "default", "id": "54a10e88-74dd-4044-985d-3140bcb34de2", "state": "normal", "checkpoint_tso": 457114216028176388, "checkpoint_time": "2025-04-04 15:39:09.961", "error": null } ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"54a10e88-74dd-4044-985d-3140bcb34de2",' '"state":' '"normal",' '"checkpoint_tso":' 457114216028176388, '"checkpoint_time":' '"2025-04-04' '15:39:09.961",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359186351336507, '"namespace":' '"default",' '"id":' '"54a10e88-74dd-4044-985d-3140bcb34de2",' '"state":' '"normal",' '"checkpoint_tso":' 457114216028176388, '"checkpoint_time":' '"2025-04-04' '15:39:09.961",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489359186351336507 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run pulsar + '[' pulsar '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 4 15:39:11 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 4 15:39:11 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster table row_format.finish_mark not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 3-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:13 CST 2025] <<<<<< run test case processor_err_chan success! >>>>>> run task successfully Changefeed remove successfully. ID: 318da0c1-e8c8-467d-bb04-1e55e00d5de8 CheckpointTs: 457114214062358534 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-8671?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7489359186351336507 457114216683798530 457114214062358534 457114216028176388 table test.finish_mark not exists for 4-th check, retry later table foreign_key.finish_mark not exists for 1-th check, retry later run task successfully table row_format.finish_mark not exists for 3-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:14 CST 2025] <<<<<< run test case csv_storage_basic success! >>>>>> Changefeed remove successfully. ID: 54a10e88-74dd-4044-985d-3140bcb34de2 CheckpointTs: 457114216945942537 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-8671?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7489359186351336507 run task successfully table test.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... table foreign_key.finish_mark not exists for 2-th check, retry later table test.finish_mark exists check table exists success check diff successfully wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... table row_format.finish_mark not exists for 4-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.17 secs (493332902 bytes/sec) [Pipeline] { [Pipeline] cache cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:17 CST 2025] <<<<<< run test case gc_safepoint success! >>>>>> wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= table test.finish_mark not exists for 2-th check, retry later [Fri Apr 4 15:39:17 CST 2025] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:39:18 CST 2025] <<<<<< run test case charset_gbk success! >>>>>> table foreign_key.finish_mark not exists for 3-th check, retry later table row_format.finish_mark not exists for 5-th check, retry later table test.finish_mark not exists for 3-th check, retry later table foreign_key.finish_mark not exists for 4-th check, retry later table row_format.finish_mark not exists for 6-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:21 CST 2025] <<<<<< run test case csv_storage_partition_table success! >>>>>> table test.finish_mark not exists for 4-th check, retry later table foreign_key.finish_mark not exists for 5-th check, retry later table row_format.finish_mark not exists for 7-th check, retry later table test.finish_mark not exists for 5-th check, retry later table foreign_key.finish_mark not exists for 6-th check, retry later table row_format.finish_mark not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... \033[0;36m<<< Run all test success >>>\033[0m table test.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct table foreign_key.finish_mark not exists for 7-th check, retry later Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table row_format.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:27 CST 2025] <<<<<< run test case canal_json_basic success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.21 secs (490653153 bytes/sec) [Pipeline] { [Pipeline] cache table foreign_key.finish_mark not exists for 8-th check, retry later table row_format.finish_mark not exists for 10-th check, retry later Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table foreign_key.finish_mark exists check diff successfully table foreign_key.t12 does not exists table row_format.finish_mark not exists for 11-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:32 CST 2025] <<<<<< run test case foreign_key success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table row_format.finish_mark exists check diff successfully table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "threadcount"="2" "mysql.user"="root" "updateproportion"="0" "readproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "dotransactions"="false" "recordcount"="10" "mysql.db"="sink_retry" "requestdistribution"="uniform" "operationcount"="0" "scanproportion"="0" "mysql.port"="4000" "readallfields"="true" "insertproportion"="0" ********************************************** Run finished, takes 4.484493ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3712.5, Avg(us): 835, Min(us): 544, Max(us): 1791, 95th(us): 2000, 99th(us): 2000 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... Verifying downstream PD is started... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:34 CST 2025] <<<<<< run test case row_format success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359394859747478 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6a118000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:4833, start at 2025-04-04 15:39:34.392019375 +0800 CST m=+1.492303925 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:34.400 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:34.393 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:34.393 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359394859747478 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6a118000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:4833, start at 2025-04-04 15:39:34.392019375 +0800 CST m=+1.492303925 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:34.400 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:34.393 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:34.393 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359399536775970 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6a4340006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:4925, start at 2025-04-04 15:39:34.550001092 +0800 CST m=+1.574833155 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:34.558 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:34.541 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:34.541 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff failed 1-th time, retry later ++ date + echo '[Fri Apr 4 15:39:37 CST 2025] <<<<<< START pulsar cluster in changefeed_reconstruct case >>>>>>' [Fri Apr 4 15:39:37 CST 2025] <<<<<< START pulsar cluster in changefeed_reconstruct case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_reconstruct + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_reconstruct + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_reconstruct/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:39:37 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_reconstruct case >>>>>>' [Fri Apr 4 15:39:37 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_reconstruct case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff successfully wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359418400075104 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6d7e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:5218, start at 2025-04-04 15:39:37.896994043 +0800 CST m=+1.565721517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:37.905 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:37.898 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:37.898 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359418400075104 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6d7e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:5218, start at 2025-04-04 15:39:37.896994043 +0800 CST m=+1.565721517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:37.905 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:37.898 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:37.898 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359414243772944 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6da1c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:5285, start at 2025-04-04 15:39:38.034958773 +0800 CST m=+1.642291251 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:38.043 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:38.041 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:38.041 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:39 CST 2025] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 8.60 secs (468489136 bytes/sec) [Pipeline] { [Pipeline] cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359412972197076 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6fb140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:4633, start at 2025-04-04 15:39:40.142900206 +0800 CST m=+1.765650229 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:40.153 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:40.151 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:40.151 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359412972197076 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6fb140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:4633, start at 2025-04-04 15:39:40.142900206 +0800 CST m=+1.765650229 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:40.153 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:40.151 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:40.151 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359426434480471 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6fd6c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:4706, start at 2025-04-04 15:39:40.279537732 +0800 CST m=+1.843579422 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:40.288 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:40.251 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:40.251 +0800 All versions after safe point can be accessed. (DO NOT EDIT) start tidb cluster in /tmp/tidb_cdc_test/canal_json_content_compatible Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ++ date + echo '[Fri Apr 4 15:39:40 CST 2025] <<<<<< START pulsar cluster in changefeed_pause_resume case >>>>>>' [Fri Apr 4 15:39:40 CST 2025] <<<<<< START pulsar cluster in changefeed_pause_resume case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_pause_resume + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_pause_resume + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_pause_resume/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_pause_resume/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:39:40 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_pause_resume case >>>>>>' [Fri Apr 4 15:39:40 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_pause_resume case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 [Fri Apr 4 15:39:42 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.59945996.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 public/default Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ++ date + echo '[Fri Apr 4 15:39:43 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:39:43 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:39:43 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.67326734.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:45 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/953f87d2-dd36-4aa0-a71b-ee844b380aee {"id":"953f87d2-dd36-4aa0-a71b-ee844b380aee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe2d5 953f87d2-dd36-4aa0-a71b-ee844b380aee /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/953f87d2-dd36-4aa0-a71b-ee844b380aee {"id":"953f87d2-dd36-4aa0-a71b-ee844b380aee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe2d5 953f87d2-dd36-4aa0-a71b-ee844b380aee /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/953f87d2-dd36-4aa0-a71b-ee844b380aee {"id":"953f87d2-dd36-4aa0-a71b-ee844b380aee","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe2d5 953f87d2-dd36-4aa0-a71b-ee844b380aee /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:39:45 CST 2025] <<<<<< START pulsar cluster in ddl_manager case >>>>>>' [Fri Apr 4 15:39:45 CST 2025] <<<<<< START pulsar cluster in ddl_manager case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_manager + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_manager + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_manager/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_manager/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_manager/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_manager/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_manager/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_manager/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_manager/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_manager/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_manager/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:39:45 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_manager case >>>>>>' [Fri Apr 4 15:39:45 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_manager case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_manager/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_manager/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_manager/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... public/default + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:46 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6c4d9870-f8f8-43b0-a5dc-82f712b92205 {"id":"6c4d9870-f8f8-43b0-a5dc-82f712b92205","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752384} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a91f8 6c4d9870-f8f8-43b0-a5dc-82f712b92205 /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6c4d9870-f8f8-43b0-a5dc-82f712b92205 {"id":"6c4d9870-f8f8-43b0-a5dc-82f712b92205","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752384} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a91f8 6c4d9870-f8f8-43b0-a5dc-82f712b92205 /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6c4d9870-f8f8-43b0-a5dc-82f712b92205 {"id":"6c4d9870-f8f8-43b0-a5dc-82f712b92205","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752384} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a91f8 6c4d9870-f8f8-43b0-a5dc-82f712b92205 /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:39:47 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:39:47 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:39:47 CST 2025] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.70547056.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:39:47 CST 2025] <<<<<< START Pulsar consumer in changefeed_reconstruct case >>>>>> ***************** properties ***************** "mysql.db"="changefeed_reconstruct" "operationcount"="0" "threadcount"="4" "readallfields"="true" "mysql.port"="4000" "dotransactions"="false" "insertproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.user"="root" "readproportion"="0" "updateproportion"="0" "recordcount"="50" "workload"="core" "scanproportion"="0" ********************************************** Run finished, takes 19.43527ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3248.3, Avg(us): 1517, Min(us): 946, Max(us): 4597, 95th(us): 5000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359448975299426 TiDB Cluster ID. tikv_gc_leader_uuid 657fef76eac0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:5947, start at 2025-04-04 15:39:47.511402008 +0800 CST m=+1.639117640 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:47.520 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:47.499 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:47.499 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359448975299426 TiDB Cluster ID. tikv_gc_leader_uuid 657fef76eac0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:5947, start at 2025-04-04 15:39:47.511402008 +0800 CST m=+1.639117640 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:47.520 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:47.499 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:47.499 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359455071837556 TiDB Cluster ID. tikv_gc_leader_uuid 657fef76de4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-pktl-4n2gs, pid:6007, start at 2025-04-04 15:39:47.476009814 +0800 CST m=+1.540029604 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:47.484 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:47.449 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:47.449 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: pulsar... <<================= Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.53 secs (535235211 bytes/sec) [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:50 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66815296-de0f-4dcf-a906-8b7ea1d864c0 {"id":"66815296-de0f-4dcf-a906-8b7ea1d864c0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752387} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd980ff4 66815296-de0f-4dcf-a906-8b7ea1d864c0 /tidb/cdc/default/default/upstream/7489359418400075104 {"id":7489359418400075104,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66815296-de0f-4dcf-a906-8b7ea1d864c0 {"id":"66815296-de0f-4dcf-a906-8b7ea1d864c0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752387} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd980ff4 66815296-de0f-4dcf-a906-8b7ea1d864c0 /tidb/cdc/default/default/upstream/7489359418400075104 {"id":7489359418400075104,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66815296-de0f-4dcf-a906-8b7ea1d864c0 {"id":"66815296-de0f-4dcf-a906-8b7ea1d864c0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752387} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd980ff4 66815296-de0f-4dcf-a906-8b7ea1d864c0 /tidb/cdc/default/default/upstream/7489359418400075104 {"id":7489359418400075104,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... [Fri Apr 4 15:39:50 CST 2025] <<<<<< START cdc server in canal_json_content_compatible case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_content_compatible.73157317.out server --log-file /tmp/tidb_cdc_test/canal_json_content_compatible/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_content_compatible/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:39:50 CST 2025] <<<<<< START Pulsar consumer in changefeed_pause_resume case >>>>>> check diff successfully public/default ++ date + echo '[Fri Apr 4 15:39:52 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:39:52 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.6463.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager wait process 6737 exit for 1-th time... wait process 6737 exit for 2-th time... table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359477834150448 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7baa0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:4760, start at 2025-04-04 15:39:52.384531926 +0800 CST m=+1.602604924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359477834150448 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7baa0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:4760, start at 2025-04-04 15:39:52.384531926 +0800 CST m=+1.602604924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359476370708349 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7bdbc0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:4840, start at 2025-04-04 15:39:52.566577889 +0800 CST m=+1.708935372 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:52.574 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:52.559 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:52.559 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (6737) - No such process wait process 6737 exit for 3-th time... process 6737 already exit check_no_capture http://127.0.0.1:2379 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:53 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/baaa0a00-59d5-41f6-b834-b7a2f2bceb0d {"id":"baaa0a00-59d5-41f6-b834-b7a2f2bceb0d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752390} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc116cd baaa0a00-59d5-41f6-b834-b7a2f2bceb0d /tidb/cdc/default/default/upstream/7489359448975299426 {"id":7489359448975299426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/baaa0a00-59d5-41f6-b834-b7a2f2bceb0d {"id":"baaa0a00-59d5-41f6-b834-b7a2f2bceb0d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752390} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc116cd baaa0a00-59d5-41f6-b834-b7a2f2bceb0d /tidb/cdc/default/default/upstream/7489359448975299426 {"id":7489359448975299426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/baaa0a00-59d5-41f6-b834-b7a2f2bceb0d {"id":"baaa0a00-59d5-41f6-b834-b7a2f2bceb0d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752390} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc116cd baaa0a00-59d5-41f6-b834-b7a2f2bceb0d /tidb/cdc/default/default/upstream/7489359448975299426 {"id":7489359448975299426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:39:53 CST 2025] <<<<<< START pulsar cluster in canal_json_content_compatible case >>>>>>' [Fri Apr 4 15:39:53 CST 2025] <<<<<< START pulsar cluster in canal_json_content_compatible case >>>>>> + workdir=/tmp/tidb_cdc_test/canal_json_content_compatible + cluster_type=normal + cd /tmp/tidb_cdc_test/canal_json_content_compatible + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/canal_json_content_compatible/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/canal_json_content_compatible/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:39:53 CST 2025] <<<<<< START pulsar cluster in normal mode in canal_json_content_compatible case >>>>>>' [Fri Apr 4 15:39:53 CST 2025] <<<<<< START pulsar cluster in normal mode in canal_json_content_compatible case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:39:53 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.70397041.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7489359412972197076,"namespace":"default","id":"ddl-manager","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:39:53.659268132+08:00","start_ts":457114227284639749,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114227284639749,"checkpoint_ts":457114227284639749,"checkpoint_time":"2025-04-04 15:39:52.901"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.6231.out cli tso query --pd=http://127.0.0.1:2379 + set +x [Fri Apr 4 15:39:55 CST 2025] <<<<<< START Pulsar consumer in ddl_manager case >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:56 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/079ae6dc-48c8-4990-98ac-0fe37895be76 {UpstreamID:7489359394859747478 Namespace:default ID:079ae6dc-48c8-4990-98ac-0fe37895be76 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:47.531241028 +0800 CST StartTs:457114225709678596 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004c1e000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114225866964998} {CheckpointTs:457114226771361798 MinTableBarrierTs:457114228082081797 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114226771361798, checkpointTs: 457114226771361798, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 {"id":"0e6a37f4-de2e-499e-b9c1-b7d8ebb60003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752393} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a9296 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 /tidb/cdc/default/default/changefeed/info/079ae6dc-48c8-4990-98ac-0fe37895be76 {"upstream-id":7489359394859747478,"namespace":"default","changefeed-id":"079ae6dc-48c8-4990-98ac-0fe37895be76","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:47.531241028+08:00","start-ts":457114225709678596,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114225866964998} /tidb/cdc/default/default/changefeed/status/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":457114226771361798,"min-table-barrier-ts":457114228082081797,"admin-job-type":0} /tidb/cdc/default/default/task/position/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/079ae6dc-48c8-4990-98ac-0fe37895be76 {UpstreamID:7489359394859747478 Namespace:default ID:079ae6dc-48c8-4990-98ac-0fe37895be76 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:47.531241028 +0800 CST StartTs:457114225709678596 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004c1e000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114225866964998} {CheckpointTs:457114226771361798 MinTableBarrierTs:457114228082081797 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114226771361798, checkpointTs: 457114226771361798, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 {"id":"0e6a37f4-de2e-499e-b9c1-b7d8ebb60003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752393} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a9296 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 /tidb/cdc/default/default/changefeed/info/079ae6dc-48c8-4990-98ac-0fe37895be76 {"upstream-id":7489359394859747478,"namespace":"default","changefeed-id":"079ae6dc-48c8-4990-98ac-0fe37895be76","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:47.531241028+08:00","start-ts":457114225709678596,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114225866964998} /tidb/cdc/default/default/changefeed/status/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":457114226771361798,"min-table-barrier-ts":457114228082081797,"admin-job-type":0} /tidb/cdc/default/default/task/position/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/079ae6dc-48c8-4990-98ac-0fe37895be76 {UpstreamID:7489359394859747478 Namespace:default ID:079ae6dc-48c8-4990-98ac-0fe37895be76 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:47.531241028 +0800 CST StartTs:457114225709678596 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004c1e000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114225866964998} {CheckpointTs:457114226771361798 MinTableBarrierTs:457114228082081797 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114226771361798, checkpointTs: 457114226771361798, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 {"id":"0e6a37f4-de2e-499e-b9c1-b7d8ebb60003","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752393} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8a9296 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 /tidb/cdc/default/default/changefeed/info/079ae6dc-48c8-4990-98ac-0fe37895be76 {"upstream-id":7489359394859747478,"namespace":"default","changefeed-id":"079ae6dc-48c8-4990-98ac-0fe37895be76","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-9964?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:47.531241028+08:00","start-ts":457114225709678596,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114225866964998} /tidb/cdc/default/default/changefeed/status/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":457114226771361798,"min-table-barrier-ts":457114228082081797,"admin-job-type":0} /tidb/cdc/default/default/task/position/0e6a37f4-de2e-499e-b9c1-b7d8ebb60003/079ae6dc-48c8-4990-98ac-0fe37895be76 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359394859747478 {"id":7489359394859747478,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id + set +x + tso='457114227968835585 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114227968835585 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:39:56 CST 2025] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.62796281.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t3 exists VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359495950569716 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7f9c4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:4684, start at 2025-04-04 15:39:56.45092198 +0800 CST m=+1.516187901 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:56.458 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:56.451 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:56.451 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359495950569716 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7f9c4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:4684, start at 2025-04-04 15:39:56.45092198 +0800 CST m=+1.516187901 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:56.458 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:56.451 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:56.451 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359496648272163 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7fb44000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:4751, start at 2025-04-04 15:39:56.517669968 +0800 CST m=+1.518661599 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:56.526 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:56.497 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:56.497 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public "id": "0e6a37f4-de2e-499e-b9c1-b7d8ebb60003", "cluster-id": "default" run task successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc) 4030545920 bytes in 7.50 secs (537470080 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G17 [Pipeline] sh + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.6192.out cli tso query --pd=http://127.0.0.1:2379 Run cases: clustered_index processor_resolved_ts_fallback GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-s05zz HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=18ddc344-bf6e-42f5-addb-328fab1be11e GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G17 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-s05zz PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:39:59 CST 2025] <<<<<< skip test case clustered_index for pulsar! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G11 [Pipeline] sh capture_id: 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 check_processor_table_count http://127.0.0.1:2379 079ae6dc-48c8-4990-98ac-0fe37895be76 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 1 public/default ++ date + echo '[Fri Apr 4 15:39:59 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:39:59 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_content_compatible.cli.7784.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-canal-json-content-compatible?protocol=canal-json&enable-tidb-extension=true' Run cases: resolve_lock move_table autorandom generate_column GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-vsw6p HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=57d0e917-8f52-407c-8747-b65016f64f65 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G11 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-vsw6p PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: pulsar... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G14 [Pipeline] sh run task successfully check_processor_table_count http://127.0.0.1:2379 079ae6dc-48c8-4990-98ac-0fe37895be76 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 0 Run cases: changefeed_finish force_replicate_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k6tq6 HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=a3de188a-6cb8-4a8d-b1f9-60cfcfbab0c9 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G14 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-k6tq6 pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G13 [Pipeline] sh table count 1 does equal to expected count 0 run task failed 1-th time, retry later Run cases: tiflash region_merge common_1 GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-wb2gs HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=db20e5aa-a724-42de-af8b-1ef75c9af84f GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G13 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-wb2gs PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G12 [Pipeline] sh + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:00 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c815add-8809-4783-9778-c2daa9d7b023 {"id":"8c815add-8809-4783-9778-c2daa9d7b023","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd476e6 8c815add-8809-4783-9778-c2daa9d7b023 /tidb/cdc/default/default/upstream/7489359477834150448 {"id":7489359477834150448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c815add-8809-4783-9778-c2daa9d7b023 {"id":"8c815add-8809-4783-9778-c2daa9d7b023","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd476e6 8c815add-8809-4783-9778-c2daa9d7b023 /tidb/cdc/default/default/upstream/7489359477834150448 {"id":7489359477834150448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c815add-8809-4783-9778-c2daa9d7b023 {"id":"8c815add-8809-4783-9778-c2daa9d7b023","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd476e6 8c815add-8809-4783-9778-c2daa9d7b023 /tidb/cdc/default/default/upstream/7489359477834150448 {"id":7489359477834150448,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:00 CST 2025] <<<<<< START pulsar cluster in ddl_puller_lag case >>>>>>' [Fri Apr 4 15:40:00 CST 2025] <<<<<< START pulsar cluster in ddl_puller_lag case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_puller_lag + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_puller_lag + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_puller_lag/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_puller_lag/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_puller_lag/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:00 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_puller_lag case >>>>>>' [Fri Apr 4 15:40:00 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_puller_lag case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_puller_lag/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_puller_lag/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_puller_lag/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-cf30g HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=fec7a446-eab7-4273-87bb-3202e5a28845 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G12 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-cf30g pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: pulsar... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G16 [Pipeline] sh Create changefeed successfully! ID: ae7083a6-23cd-4aa5-bc5c-4a87401dd53b Info: {"upstream_id":7489359448975299426,"namespace":"default","id":"ae7083a6-23cd-4aa5-bc5c-4a87401dd53b","sink_uri":"pulsar://127.0.0.1:6650/ticdc-canal-json-content-compatible?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:00.684314883+08:00","start_ts":457114229171552258,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114229171552258,"checkpoint_ts":457114229171552258,"checkpoint_time":"2025-04-04 15:40:00.099"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Run cases: owner_resign processor_etcd_worker_delay sink_hang GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-xz56z HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=1c317660-7faa-4e27-9daf-28a7d52f13bb GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G16 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-xz56z pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G15 [Pipeline] // timeout [Pipeline] } + set +x + tso='457114229054111745 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114229054111745 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:01 CST 2025] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.62236225.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Run cases: new_ci_collation batch_add_table multi_rocks ci_collation_compatibility GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6cb1w HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=06455efb-8048-4eae-a116-02f0e0a27d9b GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G15 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-6cb1w PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: pulsar... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G10 Run cases: default_value simple cdc_server_tips event_filter sql_mode GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_pulsar_test","buildid":"1908057860684648450","prowjobid":"4abd5cac-c003-428e-ab26-3bfc2e2dc49a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12120,"author":"dveeden","sha":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","title":"CI: Upgrade to actions/cache@v4","head_ref":"ci_actions_cache_v4","link":"https://github.com/pingcap/tiflow/pull/12120","commit_link":"https://github.com/pingcap/tiflow/pull/12120/commits/e8bfbe4ba98e05140ab733ee3d08dd3b99918721","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908057860684648450 POD_LABEL=pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-bmrsl HOSTNAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=changes JENKINS_NODE_COOKIE=9948f164-7a40-4152-bc0e-9420049f7b4d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_pulsar_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3485 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_pulsar_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/display/redirect PROW_JOB_ID=4abd5cac-c003-428e-ab26-3bfc2e2dc49a TEST_GROUP=G10 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/3485/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_pulsar_test/ BUILD_DISPLAY_NAME=#3485 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3 pingcap_tiflow_pull_cdc_integration_pulsar_test_3485-bmrsl PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-1c247575f7a5458f3a8df09baf43395940069f8be33f18a8fa953544ffab3376 _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff failed 1-th time, retry later + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check_processor_table_count http://127.0.0.1:2379 079ae6dc-48c8-4990-98ac-0fe37895be76 0e6a37f4-de2e-499e-b9c1-b7d8ebb60003 0 run task successfully ***************** properties ***************** "mysql.user"="root" "readallfields"="true" "mysql.port"="4000" "dotransactions"="false" "operationcount"="0" "scanproportion"="0" "mysql.db"="changefeed_reconstruct" "workload"="core" "readproportion"="0" "updateproportion"="0" "threadcount"="4" "recordcount"="50" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "insertproportion"="0" ********************************************** Run finished, takes 18.845395ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3260.9, Avg(us): 1396, Min(us): 950, Max(us): 6455, 95th(us): 5000, 99th(us): 7000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 2-th time, retry later The 1 times to try to start tidb cluster... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:04 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/090dd56b-c65c-49e4-a249-d5df5324d974 {"id":"090dd56b-c65c-49e4-a249-d5df5324d974","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752401} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbde4a2d8 090dd56b-c65c-49e4-a249-d5df5324d974 /tidb/cdc/default/default/upstream/7489359495950569716 {"id":7489359495950569716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/090dd56b-c65c-49e4-a249-d5df5324d974 {"id":"090dd56b-c65c-49e4-a249-d5df5324d974","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752401} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbde4a2d8 090dd56b-c65c-49e4-a249-d5df5324d974 /tidb/cdc/default/default/upstream/7489359495950569716 {"id":7489359495950569716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/090dd56b-c65c-49e4-a249-d5df5324d974 {"id":"090dd56b-c65c-49e4-a249-d5df5324d974","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752401} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbde4a2d8 090dd56b-c65c-49e4-a249-d5df5324d974 /tidb/cdc/default/default/upstream/7489359495950569716 {"id":7489359495950569716,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:04 CST 2025] <<<<<< START pulsar cluster in drop_many_tables case >>>>>>' [Fri Apr 4 15:40:04 CST 2025] <<<<<< START pulsar cluster in drop_many_tables case >>>>>> + workdir=/tmp/tidb_cdc_test/drop_many_tables + cluster_type=normal + cd /tmp/tidb_cdc_test/drop_many_tables + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/drop_many_tables/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/drop_many_tables/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/drop_many_tables/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/drop_many_tables/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/drop_many_tables/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/drop_many_tables/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/drop_many_tables/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/drop_many_tables/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/drop_many_tables/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:04 CST 2025] <<<<<< START pulsar cluster in normal mode in drop_many_tables case >>>>>>' [Fri Apr 4 15:40:04 CST 2025] <<<<<< START pulsar cluster in normal mode in drop_many_tables case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/drop_many_tables/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/drop_many_tables/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/drop_many_tables/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table changefeed_reconstruct.usertable exists VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359525200891492 TiDB Cluster ID. tikv_gc_leader_uuid 657fef86964000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:4940, start at 2025-04-04 15:40:03.576224885 +0800 CST m=+1.445804888 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:03.584 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:03.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:03.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359525200891492 TiDB Cluster ID. tikv_gc_leader_uuid 657fef86964000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:4940, start at 2025-04-04 15:40:03.576224885 +0800 CST m=+1.445804888 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:03.584 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:03.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:03.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359523581769157 TiDB Cluster ID. tikv_gc_leader_uuid 657fef86bb4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-n503-1b490, pid:5015, start at 2025-04-04 15:40:03.714006238 +0800 CST m=+1.519563719 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:03.723 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:03.693 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:03.693 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:02 CST 2025] <<<<<< run test case sink_retry success! >>>>>> check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.6415.out cli tso query --pd=http://127.0.0.1:2379 public/default ++ date + echo '[Fri Apr 4 15:40:06 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:06 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.6749.out cli changefeed create --start-ts=457114227968835585 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-25448?protocol=canal-json&enable-tidb-extension=true' check diff failed 4-th time, retry later Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully wait process cdc.test exit for 1-th time... [Fri Apr 4 15:40:07 CST 2025] <<<<<< START Pulsar consumer in canal_json_content_compatible case >>>>>> [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) Create changefeed successfully! ID: 893acf84-22a3-47ed-a8fc-0121a6998565 Info: {"upstream_id":7489359477834150448,"namespace":"default","id":"893acf84-22a3-47ed-a8fc-0121a6998565","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-25448?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:07.480464903+08:00","start_ts":457114227968835585,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114227968835585,"checkpoint_ts":457114227968835585,"checkpoint_time":"2025-04-04 15:39:55.511"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... + set +x + tso='457114230939975681 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114230939975681 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:08 CST 2025] <<<<<< run test case changefeed_reconstruct success! >>>>>> Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + set +x [Fri Apr 4 15:40:08 CST 2025] <<<<<< START Pulsar consumer in ddl_puller_lag case >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Fri Apr 4 15:40:09 CST 2025] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.64506452.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' Starting Upstream TiDB... + '[' 0 -eq 50 ']' + sleep 3 public/default Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ++ date + echo '[Fri Apr 4 15:40:10 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:10 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.6697.out cli changefeed create --start-ts=457114229054111745 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-drop-tables-test-11726?protocol=canal-json&enable-tidb-extension=true' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359553081123144 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8cdac0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:1483, start at 2025-04-04 15:40:09.96891966 +0800 CST m=+1.524726056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:09.977 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:09.963 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:09.963 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359553081123144 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8cdac0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:1483, start at 2025-04-04 15:40:09.96891966 +0800 CST m=+1.524726056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:09.977 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:09.963 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:09.963 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359553616360779 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8ce68000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:1554, start at 2025-04-04 15:40:10.039367702 +0800 CST m=+1.515460618 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.047 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.010 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.010 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359548330505714 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d89c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r, pid:1405, start at 2025-04-04 15:40:10.693305864 +0800 CST m=+1.586194584 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.701 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.663 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.663 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359548330505714 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d89c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r, pid:1405, start at 2025-04-04 15:40:10.693305864 +0800 CST m=+1.586194584 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.701 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.663 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.663 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359558683523792 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d9540004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-s05z-f3s6r, pid:1473, start at 2025-04-04 15:40:10.713870001 +0800 CST m=+1.530782942 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.721 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.709 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.709 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 34eff47b-5155-4571-a0d5-69c8c6cfe4e8 Info: {"upstream_id":7489359495950569716,"namespace":"default","id":"34eff47b-5155-4571-a0d5-69c8c6cfe4e8","sink_uri":"pulsar://127.0.0.1:6650/ticdc-drop-tables-test-11726?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:11.572699498+08:00","start_ts":457114229054111745,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114229054111745,"checkpoint_ts":457114229054111745,"checkpoint_time":"2025-04-04 15:39:59.651"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359554550992290 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e3d80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:1345, start at 2025-04-04 15:40:11.397243492 +0800 CST m=+1.510106842 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.405 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.382 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.382 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359554550992290 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e3d80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:1345, start at 2025-04-04 15:40:11.397243492 +0800 CST m=+1.510106842 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.405 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.382 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.382 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359556792121542 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e3d4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:1413, start at 2025-04-04 15:40:11.430091158 +0800 CST m=+1.473831622 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.438 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... [Pipeline] // cache Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359562039084656 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e8a40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:1338, start at 2025-04-04 15:40:11.727426907 +0800 CST m=+1.560825579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.736 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359562039084656 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e8a40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:1338, start at 2025-04-04 15:40:11.727426907 +0800 CST m=+1.560825579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.736 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359556372101488 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8e8ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:1417, start at 2025-04-04 15:40:11.73461333 +0800 CST m=+1.492207997 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:11.742 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:11.741 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:11.741 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2870.out cli tso query --pd=http://127.0.0.1:2379 + set +x [Fri Apr 4 15:40:13 CST 2025] <<<<<< START Pulsar consumer in drop_many_tables case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:12 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e1d3132-95dd-4196-adb0-124fc3f245d7 {"id":"2e1d3132-95dd-4196-adb0-124fc3f245d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdfec9fe 2e1d3132-95dd-4196-adb0-124fc3f245d7 /tidb/cdc/default/default/upstream/7489359525200891492 {"id":7489359525200891492,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e1d3132-95dd-4196-adb0-124fc3f245d7 {"id":"2e1d3132-95dd-4196-adb0-124fc3f245d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdfec9fe 2e1d3132-95dd-4196-adb0-124fc3f245d7 /tidb/cdc/default/default/upstream/7489359525200891492 {"id":7489359525200891492,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e1d3132-95dd-4196-adb0-124fc3f245d7 {"id":"2e1d3132-95dd-4196-adb0-124fc3f245d7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdfec9fe 2e1d3132-95dd-4196-adb0-124fc3f245d7 /tidb/cdc/default/default/upstream/7489359525200891492 {"id":7489359525200891492,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:12 CST 2025] <<<<<< START pulsar cluster in split_region case >>>>>>' [Fri Apr 4 15:40:12 CST 2025] <<<<<< START pulsar cluster in split_region case >>>>>> + workdir=/tmp/tidb_cdc_test/split_region + cluster_type=oauth + cd /tmp/tidb_cdc_test/split_region + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/split_region/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/split_region/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/split_region/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/split_region/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/split_region/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/split_region/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/split_region/pulsar_standalone.conf + pulsar_port=6650 + '[' oauth == mtls ']' + '[' oauth == oauth ']' + echo 'Waiting for oauth2 server to be ready...' Waiting for oauth2 server to be ready... + i=0 + nc -z localhost 9096 + oauth2-server + gen_oauth_config + echo ' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ date + echo '[Fri Apr 4 15:40:12 CST 2025] <<<<<< START pulsar cluster in oauth mode in split_region case >>>>>>' [Fri Apr 4 15:40:12 CST 2025] <<<<<< START pulsar cluster in oauth mode in split_region case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/split_region/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/split_region/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/split_region/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withEnv [Pipeline] } check diff failed 1-th time, retry later [Pipeline] // timeout [Pipeline] } [Pipeline] // node [Pipeline] } [Fri Apr 4 15:40:13 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.28242826.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // stage [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359566708424978 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fd2c003c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:1568, start at 2025-04-04 15:40:13.028971265 +0800 CST m=+1.483209131 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:13.035 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:13.003 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:13.003 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359566708424978 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fd2c003c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:1568, start at 2025-04-04 15:40:13.028971265 +0800 CST m=+1.483209131 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:13.035 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:13.003 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:13.003 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359567384406734 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fed0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:1639, start at 2025-04-04 15:40:13.13579055 +0800 CST m=+1.515492716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:13.143 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:13.108 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:13.108 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // container ++ date + echo '[Fri Apr 4 15:40:14 CST 2025] <<<<<< START pulsar cluster in changefeed_finish case >>>>>>' [Fri Apr 4 15:40:14 CST 2025] <<<<<< START pulsar cluster in changefeed_finish case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_finish + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_finish + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_finish/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_finish/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_finish/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_finish/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_finish/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_finish/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_finish/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_finish/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_finish/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:14 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_finish case >>>>>>' [Fri Apr 4 15:40:14 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_finish case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_finish/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_finish/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_finish/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2881.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] } [Pipeline] // podTemplate [Pipeline] } + set +x + tso='457114232596463617 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114232596463617 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:14 CST 2025] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.29132915.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359569710246327 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9108c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:1503, start at 2025-04-04 15:40:14.257819644 +0800 CST m=+1.588714834 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:14.267 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:14.243 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:14.243 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359569710246327 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9108c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:1503, start at 2025-04-04 15:40:14.257819644 +0800 CST m=+1.588714834 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:14.267 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:14.243 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:14.243 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359565526853422 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9108c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:1576, start at 2025-04-04 15:40:14.259219007 +0800 CST m=+1.516850378 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:14.267 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:14.243 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:14.243 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // stage [Pipeline] } + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2957.out cli tso query --pd=http://127.0.0.1:2379 start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > + set +x + tso='457114233061769217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114233061769217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:16 CST 2025] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.29242926.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:16 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:16 CST 2025] <<<<<< START pulsar cluster in processor_resolved_ts_fallback case >>>>>>' [Fri Apr 4 15:40:16 CST 2025] <<<<<< START pulsar cluster in processor_resolved_ts_fallback case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_resolved_ts_fallback + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:16 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_resolved_ts_fallback case >>>>>>' [Fri Apr 4 15:40:16 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_resolved_ts_fallback case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359569257874183 TiDB Cluster ID. tikv_gc_leader_uuid 657fef92d24000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:1425, start at 2025-04-04 15:40:16.091645789 +0800 CST m=+1.569675116 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:16.100 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:16.073 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:16.073 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359569257874183 TiDB Cluster ID. tikv_gc_leader_uuid 657fef92d24000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:1425, start at 2025-04-04 15:40:16.091645789 +0800 CST m=+1.569675116 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:16.100 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:16.073 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:16.073 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359579378854246 TiDB Cluster ID. tikv_gc_leader_uuid 657fef92dfc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:1499, start at 2025-04-04 15:40:16.143065048 +0800 CST m=+1.547032865 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:16.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:16.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:16.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table drop_tables.c not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ date + echo '[Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in capture_session_done_during_task case >>>>>>' [Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in capture_session_done_during_task case >>>>>> + workdir=/tmp/tidb_cdc_test/capture_session_done_during_task + cluster_type=normal + cd /tmp/tidb_cdc_test/capture_session_done_during_task + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/capture_session_done_during_task/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/capture_session_done_during_task/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in normal mode in capture_session_done_during_task case >>>>>>' [Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in normal mode in capture_session_done_during_task case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff failed 3-th time, retry later + set +x + tso='457114233406488577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114233406488577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:17 CST 2025] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.29902992.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:17 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce36cc0a-1b13-4164-83db-8d7fc12e9ffb {"id":"ce36cc0a-1b13-4164-83db-8d7fc12e9ffb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1b5fd5 ce36cc0a-1b13-4164-83db-8d7fc12e9ffb /tidb/cdc/default/default/upstream/7489359553081123144 {"id":7489359553081123144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce36cc0a-1b13-4164-83db-8d7fc12e9ffb {"id":"ce36cc0a-1b13-4164-83db-8d7fc12e9ffb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1b5fd5 ce36cc0a-1b13-4164-83db-8d7fc12e9ffb /tidb/cdc/default/default/upstream/7489359553081123144 {"id":7489359553081123144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce36cc0a-1b13-4164-83db-8d7fc12e9ffb {"id":"ce36cc0a-1b13-4164-83db-8d7fc12e9ffb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752414} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1b5fd5 ce36cc0a-1b13-4164-83db-8d7fc12e9ffb /tidb/cdc/default/default/upstream/7489359553081123144 {"id":7489359553081123144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in tiflash case >>>>>>' [Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in tiflash case >>>>>> + workdir=/tmp/tidb_cdc_test/tiflash + cluster_type=mtls + cd /tmp/tidb_cdc_test/tiflash + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/tiflash/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/tiflash/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/tiflash/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/tiflash/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/tiflash/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ...................................................................+++++ ..................................+++++ e is 65537 (0x010001) + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ...........+++++ ...............................................+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=CN = server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ........+++++ ..................+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=CN = broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/tiflash/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/tiflash/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem' ++ date + echo '[Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in mtls mode in tiflash case >>>>>>' [Fri Apr 4 15:40:17 CST 2025] <<<<<< START pulsar cluster in mtls mode in tiflash case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/tiflash/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/tiflash/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/tiflash/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 [Fri Apr 4 15:40:18 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.29652967.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 public/default ++ date + echo '[Fri Apr 4 15:40:19 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:19 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.6919.out cli changefeed create --start-ts=457114230939975681 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-split-region-test-14165?protocol=canal-json&enable-tidb-extension=true' --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml table drop_tables.c not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:19 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da8d86eb-8e9b-4498-872a-3573a2741c38 {"id":"da8d86eb-8e9b-4498-872a-3573a2741c38","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752416} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe202ae8 da8d86eb-8e9b-4498-872a-3573a2741c38 /tidb/cdc/default/default/upstream/7489359562039084656 {"id":7489359562039084656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da8d86eb-8e9b-4498-872a-3573a2741c38 {"id":"da8d86eb-8e9b-4498-872a-3573a2741c38","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752416} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe202ae8 da8d86eb-8e9b-4498-872a-3573a2741c38 /tidb/cdc/default/default/upstream/7489359562039084656 {"id":7489359562039084656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da8d86eb-8e9b-4498-872a-3573a2741c38 {"id":"da8d86eb-8e9b-4498-872a-3573a2741c38","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752416} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe202ae8 da8d86eb-8e9b-4498-872a-3573a2741c38 /tidb/cdc/default/default/upstream/7489359562039084656 {"id":7489359562039084656,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:19 CST 2025] <<<<<< START pulsar cluster in simple case >>>>>>' [Fri Apr 4 15:40:19 CST 2025] <<<<<< START pulsar cluster in simple case >>>>>> + workdir=/tmp/tidb_cdc_test/simple + cluster_type=normal + cd /tmp/tidb_cdc_test/simple + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/simple/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/simple/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/simple/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/simple/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/simple/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/simple/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/simple/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/simple/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/simple/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:19 CST 2025] <<<<<< START pulsar cluster in normal mode in simple case >>>>>>' [Fri Apr 4 15:40:19 CST 2025] <<<<<< START pulsar cluster in normal mode in simple case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/simple/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/simple/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/simple/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6651 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff failed 4-th time, retry later Create changefeed successfully! ID: 375ed5cb-a75e-4dc5-bb08-865305107e98 Info: {"upstream_id":7489359525200891492,"namespace":"default","id":"375ed5cb-a75e-4dc5-bb08-865305107e98","sink_uri":"pulsar://127.0.0.1:6650/ticdc-split-region-test-14165?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:20.463413217+08:00","start_ts":457114230939975681,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"oauth2":{"oauth2-issuer-url":"http://localhost:9096","oauth2-audience":"cdc-api-uri","oauth2-private-key":"/tmp/tidb_cdc_test/split_region/credential.json","oauth2-client-id":"1234"}},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114230939975681,"checkpoint_ts":457114230939975681,"checkpoint_time":"2025-04-04 15:40:06.845"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... public/default ++ date + echo '[Fri Apr 4 15:40:20 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:20 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:40:20 CST 2025] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.32993301.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:20 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/473a4089-523e-474e-9d24-c6ebd1752439 {"id":"473a4089-523e-474e-9d24-c6ebd1752439","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2772d9 473a4089-523e-474e-9d24-c6ebd1752439 /tidb/cdc/default/default/upstream/7489359566708424978 {"id":7489359566708424978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/473a4089-523e-474e-9d24-c6ebd1752439 {"id":"473a4089-523e-474e-9d24-c6ebd1752439","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2772d9 473a4089-523e-474e-9d24-c6ebd1752439 /tidb/cdc/default/default/upstream/7489359566708424978 {"id":7489359566708424978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/473a4089-523e-474e-9d24-c6ebd1752439 {"id":"473a4089-523e-474e-9d24-c6ebd1752439","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2772d9 473a4089-523e-474e-9d24-c6ebd1752439 /tidb/cdc/default/default/upstream/7489359566708424978 {"id":7489359566708424978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:20 CST 2025] <<<<<< START pulsar cluster in resolve_lock case >>>>>>' [Fri Apr 4 15:40:20 CST 2025] <<<<<< START pulsar cluster in resolve_lock case >>>>>> + workdir=/tmp/tidb_cdc_test/resolve_lock + cluster_type=normal + cd /tmp/tidb_cdc_test/resolve_lock + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/resolve_lock/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/resolve_lock/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/resolve_lock/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/resolve_lock/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/resolve_lock/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/resolve_lock/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/resolve_lock/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/resolve_lock/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/resolve_lock/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:20 CST 2025] <<<<<< START pulsar cluster in normal mode in resolve_lock case >>>>>>' [Fri Apr 4 15:40:20 CST 2025] <<<<<< START pulsar cluster in normal mode in resolve_lock case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/resolve_lock/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/resolve_lock/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/resolve_lock/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table drop_tables.c not exists for 3-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + set +x [Fri Apr 4 15:40:21 CST 2025] <<<<<< START Pulsar consumer in split_region case >>>>>> table split_region.test1 not exists for 1-th check, retry later wait process 5999 exit for 1-th time... wait process 5999 exit for 2-th time... wait process 5999 exit for 3-th time... wait process 5999 exit for 4-th time... wait process 5999 exit for 5-th time... wait process 5999 exit for 6-th time... wait process 5999 exit for 7-th time... wait process 5999 exit for 8-th time... wait process 5999 exit for 9-th time... check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:22 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2797a2ca-cbf5-48f9-9f21-cabda964a546 {"id":"2797a2ca-cbf5-48f9-9f21-cabda964a546","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2cfdd7 2797a2ca-cbf5-48f9-9f21-cabda964a546 /tidb/cdc/default/default/upstream/7489359569257874183 {"id":7489359569257874183,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2797a2ca-cbf5-48f9-9f21-cabda964a546 {"id":"2797a2ca-cbf5-48f9-9f21-cabda964a546","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2cfdd7 2797a2ca-cbf5-48f9-9f21-cabda964a546 /tidb/cdc/default/default/upstream/7489359569257874183 {"id":7489359569257874183,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2797a2ca-cbf5-48f9-9f21-cabda964a546 {"id":"2797a2ca-cbf5-48f9-9f21-cabda964a546","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2cfdd7 2797a2ca-cbf5-48f9-9f21-cabda964a546 /tidb/cdc/default/default/upstream/7489359569257874183 {"id":7489359569257874183,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:22 CST 2025] <<<<<< START pulsar cluster in batch_add_table case >>>>>>' [Fri Apr 4 15:40:22 CST 2025] <<<<<< START pulsar cluster in batch_add_table case >>>>>> + workdir=/tmp/tidb_cdc_test/batch_add_table + cluster_type=normal + cd /tmp/tidb_cdc_test/batch_add_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/batch_add_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/batch_add_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat table test.finish_mark not exists for 1-th check, retry later + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:22 CST 2025] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>>' [Fri Apr 4 15:40:22 CST 2025] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-bookie + nc -z localhost 6650 + i=1 + '[' 1 -gt 20 ']' + sleep 2 /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5999) - No such process wait process 5999 exit for 10-th time... process 5999 already exit [Fri Apr 4 15:40:22 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.66266628.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359599843045520 TiDB Cluster ID. tikv_gc_leader_uuid 657fef98158000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:4804, start at 2025-04-04 15:40:21.492886207 +0800 CST m=+1.596337328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:21.502 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:21.462 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:21.462 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359599843045520 TiDB Cluster ID. tikv_gc_leader_uuid 657fef98158000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:4804, start at 2025-04-04 15:40:21.492886207 +0800 CST m=+1.596337328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:21.502 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:21.462 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:21.462 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359602096329604 TiDB Cluster ID. tikv_gc_leader_uuid 657fef98224000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:4874, start at 2025-04-04 15:40:21.56160458 +0800 CST m=+1.597931165 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:21.569 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:21.564 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:21.564 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 public/default ++ date + echo '[Fri Apr 4 15:40:23 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:23 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.3290.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359564231334421 TiDB Cluster ID. tikv_gc_leader_uuid 657fef98e140009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:903, start at 2025-04-04 15:40:22.324884378 +0800 CST m=+11.305857035 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:22.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:22.428 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:22.428 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359564231334421 TiDB Cluster ID. tikv_gc_leader_uuid 657fef98e140009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:903, start at 2025-04-04 15:40:22.324884378 +0800 CST m=+11.305857035 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:22.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:22.428 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:22.428 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359562575665196 TiDB Cluster ID. tikv_gc_leader_uuid 657fef99344000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:976, start at 2025-04-04 15:40:22.653381822 +0800 CST m=+11.557731636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:22.661 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:22.659 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:22.659 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log table drop_tables.c exists check diff successfully + nc -z localhost 6650 arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public public/default ++ date + echo '[Fri Apr 4 15:40:23 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:23 CST 2025] <<<<<< pulsar is ready >>>>>> + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.3373.out cli tso query --pd=http://127.0.0.1:2379 Create changefeed successfully! ID: 5802fee1-da5a-49de-a9a5-fe85cc939ead Info: {"upstream_id":7489359548330505714,"namespace":"default","id":"5802fee1-da5a-49de-a9a5-fe85cc939ead","sink_uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:23.913639834+08:00","start_ts":457114235256700934,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114235256700934,"checkpoint_ts":457114235256700934,"checkpoint_time":"2025-04-04 15:40:23.312"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:23 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4cc13ba8-880c-4821-8746-adaaab3ce481 {"id":"4cc13ba8-880c-4821-8746-adaaab3ce481","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752421} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1ef4f8 4cc13ba8-880c-4821-8746-adaaab3ce481 /tidb/cdc/default/default/upstream/7489359554550992290 {"id":7489359554550992290,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4cc13ba8-880c-4821-8746-adaaab3ce481 {"id":"4cc13ba8-880c-4821-8746-adaaab3ce481","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752421} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1ef4f8 4cc13ba8-880c-4821-8746-adaaab3ce481 /tidb/cdc/default/default/upstream/7489359554550992290 {"id":7489359554550992290,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4cc13ba8-880c-4821-8746-adaaab3ce481 {"id":"4cc13ba8-880c-4821-8746-adaaab3ce481","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752421} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1ef4f8 4cc13ba8-880c-4821-8746-adaaab3ce481 /tidb/cdc/default/default/upstream/7489359554550992290 {"id":7489359554550992290,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table test.finish_mark not exists for 2-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6172.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... public/default wait process cdc.test exit for 2-th time... [Fri Apr 4 15:40:24 CST 2025] <<<<<< START Pulsar consumer in changefeed_finish case >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ++ date + echo '[Fri Apr 4 15:40:24 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:24 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:40:24 CST 2025] <<<<<< run test case drop_many_tables success! >>>>>> check diff failed 1-th time, retry later + set +x [Fri Apr 4 15:40:25 CST 2025] <<<<<< START Pulsar consumer in processor_resolved_ts_fallback case >>>>>> cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later [Fri Apr 4 15:40:25 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.28722874.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: a68690c8-fb88-4651-aeb1-94808445689f Info: {"upstream_id":7489359553081123144,"namespace":"default","id":"a68690c8-fb88-4651-aeb1-94808445689f","sink_uri":"pulsar+ssl://127.0.0.1:6651/ticdc-tiflash-test-22259?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:25.55034463+08:00","start_ts":457114232596463617,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"tls-trust-certs-file-path":"/tmp/tidb_cdc_test/tiflash/ca.cert.pem","connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"auth-tls-certificate-path":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","auth-tls-private-key-path":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114232596463617,"checkpoint_ts":457114232596463617,"checkpoint_time":"2025-04-04 15:40:13.164"} [Fri Apr 4 15:40:25 CST 2025] <<<<<< START Pulsar consumer in tiflash case >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > public/default + set +x + tso='457114235474542593 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114235474542593 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:25 CST 2025] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.34113413.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.finish_mark not exists for 3-th check, retry later check diff failed 1-th time, retry later table split_region.test2 exists check diff failed 1-th time, retry later ++ date + echo '[Fri Apr 4 15:40:26 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:26 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.3392.out cli changefeed create --start-ts=457114233061769217 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-simple-test-10970?protocol=canal-json&enable-tidb-extension=true' + set +x + tso='457114235597750273 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114235597750273 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public public/default check diff failed 2-th time, retry later Create changefeed successfully! ID: 2dbcf539-78b4-49bf-a367-30812c91dfd3 Info: {"upstream_id":7489359562039084656,"namespace":"default","id":"2dbcf539-78b4-49bf-a367-30812c91dfd3","sink_uri":"pulsar://127.0.0.1:6650/ticdc-simple-test-10970?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:27.070637315+08:00","start_ts":457114233061769217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114233061769217,"checkpoint_ts":457114233061769217,"checkpoint_time":"2025-04-04 15:40:14.939"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ***************** properties ***************** "readproportion"="0" "workload"="core" "dotransactions"="false" "readallfields"="true" "mysql.port"="4000" "requestdistribution"="uniform" "scanproportion"="0" "threadcount"="4" "updateproportion"="0" "operationcount"="0" "recordcount"="20" "mysql.db"="changefeed_error" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" ********************************************** Run finished, takes 11.24925ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3559.6, Avg(us): 2117, Min(us): 1048, Max(us): 5544, 95th(us): 6000, 99th(us): 6000 [Fri Apr 4 15:40:27 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.62366238.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Fri Apr 4 15:40:27 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:27 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.3452.out cli changefeed create --start-ts=457114233406488577 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-6992?protocol=canal-json&enable-tidb-extension=true' table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table processor_resolved_ts_fallback.t1 exists [Fri Apr 4 15:40:27 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.34283430.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228267679764,"min-table-barrier-ts":457114228267679764,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228267679764,"min-table-barrier-ts":457114228267679764,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228267679764,"min-table-barrier-ts":457114228267679764,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:40:28 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> Create changefeed successfully! ID: 4543fc9d-2554-4c53-8a02-702c405db289 Info: {"upstream_id":7489359566708424978,"namespace":"default","id":"4543fc9d-2554-4c53-8a02-702c405db289","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-6992?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:28.143697836+08:00","start_ts":457114233406488577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114233406488577,"checkpoint_ts":457114233406488577,"checkpoint_time":"2025-04-04 15:40:16.254"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.66906692.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > table test.finish_mark not exists for 4-th check, retry later + set +x [Fri Apr 4 15:40:28 CST 2025] <<<<<< START Pulsar consumer in simple case >>>>>> succeed to verify meta placement rules public/default ++ date + echo '[Fri Apr 4 15:40:28 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:28 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.3439.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-20070?protocol=canal-json&enable-tidb-extension=true' check diff successfully ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:28 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully ++ date + echo '[Fri Apr 4 15:40:28 CST 2025] <<<<<< START pulsar cluster in owner_resign case >>>>>>' [Fri Apr 4 15:40:28 CST 2025] <<<<<< START pulsar cluster in owner_resign case >>>>>> + workdir=/tmp/tidb_cdc_test/owner_resign + cluster_type=normal + cd /tmp/tidb_cdc_test/owner_resign + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/owner_resign/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/owner_resign/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/owner_resign/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/owner_resign/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/owner_resign/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/owner_resign/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/owner_resign/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/owner_resign/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/owner_resign/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:28 CST 2025] <<<<<< START pulsar cluster in normal mode in owner_resign case >>>>>>' [Fri Apr 4 15:40:28 CST 2025] <<<<<< START pulsar cluster in normal mode in owner_resign case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/owner_resign/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/owner_resign/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/owner_resign/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:28 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12766e25-e367-4612-8443-4a95bd66f210 {"id":"12766e25-e367-4612-8443-4a95bd66f210","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752426} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe29f00f 12766e25-e367-4612-8443-4a95bd66f210 /tidb/cdc/default/default/upstream/7489359569710246327 {"id":7489359569710246327,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12766e25-e367-4612-8443-4a95bd66f210 {"id":"12766e25-e367-4612-8443-4a95bd66f210","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752426} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe29f00f 12766e25-e367-4612-8443-4a95bd66f210 /tidb/cdc/default/default/upstream/7489359569710246327 {"id":7489359569710246327,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12766e25-e367-4612-8443-4a95bd66f210 {"id":"12766e25-e367-4612-8443-4a95bd66f210","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752426} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe29f00f 12766e25-e367-4612-8443-4a95bd66f210 /tidb/cdc/default/default/upstream/7489359569710246327 {"id":7489359569710246327,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later Create changefeed successfully! ID: 83947fdd-07a1-4814-8fb6-a5433650aa70 Info: {"upstream_id":7489359569257874183,"namespace":"default","id":"83947fdd-07a1-4814-8fb6-a5433650aa70","sink_uri":"pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-20070?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:29.432319745+08:00","start_ts":457114236714483715,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114236714483715,"checkpoint_ts":457114236714483715,"checkpoint_time":"2025-04-04 15:40:28.873"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff successfully + set +x [Fri Apr 4 15:40:29 CST 2025] <<<<<< START Pulsar consumer in resolve_lock case >>>>>> go: downloading go1.23.4 (linux/amd64) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359631824678661 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9f8c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:8455, start at 2025-04-04 15:40:29.147670886 +0800 CST m=+1.485552023 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:29.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:29.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:29.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359631824678661 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9f8c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:8455, start at 2025-04-04 15:40:29.147670886 +0800 CST m=+1.485552023 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:29.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:29.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:29.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359633921041428 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9fbd80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:8531, start at 2025-04-04 15:40:29.309506455 +0800 CST m=+1.570991705 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:29.319 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:29.302 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:29.302 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 3-th time, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:30 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b {"id":"b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0dea b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b {"id":"b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0dea b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b {"id":"b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0dea b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:30 CST 2025] <<<<<< START pulsar cluster in changefeed_error case >>>>>>' [Fri Apr 4 15:40:30 CST 2025] <<<<<< START pulsar cluster in changefeed_error case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_error + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_error + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_error/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_error/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_error/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_error/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_error/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_error/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_error/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_error/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_error/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:30 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_error case >>>>>>' [Fri Apr 4 15:40:30 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_error case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_error/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_error/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_error/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/5802fee1-da5a-49de-a9a5-fe85cc939ead {UpstreamID:7489359548330505714 Namespace:default ID:5802fee1-da5a-49de-a9a5-fe85cc939ead SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:23.913639834 +0800 CST StartTs:457114235256700934 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba7950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114235414249473} {CheckpointTs:457114237052387332 MinTableBarrierTs:457114237052387332 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/capture/841af78d-d649-4521-b501-2fdee5c1fa33 {"id":"841af78d-d649-4521-b501-2fdee5c1fa33","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180b77 841af78d-d649-4521-b501-2fdee5c1fa33 /tidb/cdc/default/default/changefeed/info/5802fee1-da5a-49de-a9a5-fe85cc939ead {"upstream-id":7489359548330505714,"namespace":"default","changefeed-id":"5802fee1-da5a-49de-a9a5-fe85cc939ead","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:23.913639834+08:00","start-ts":457114235256700934,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114235414249473} /tidb/cdc/default/default/changefeed/status/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":457114237052387332,"min-table-barrier-ts":457114237052387332,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e537199-dad7-4a9e-873f-624cab54beea/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/841af78d-d649-4521-b501-2fdee5c1fa33/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/5802fee1-da5a-49de-a9a5-fe85cc939ead {UpstreamID:7489359548330505714 Namespace:default ID:5802fee1-da5a-49de-a9a5-fe85cc939ead SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:23.913639834 +0800 CST StartTs:457114235256700934 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba7950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114235414249473} {CheckpointTs:457114237052387332 MinTableBarrierTs:457114237052387332 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/capture/841af78d-d649-4521-b501-2fdee5c1fa33 {"id":"841af78d-d649-4521-b501-2fdee5c1fa33","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180b77 841af78d-d649-4521-b501-2fdee5c1fa33 /tidb/cdc/default/default/changefeed/info/5802fee1-da5a-49de-a9a5-fe85cc939ead {"upstream-id":7489359548330505714,"namespace":"default","changefeed-id":"5802fee1-da5a-49de-a9a5-fe85cc939ead","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:23.913639834+08:00","start-ts":457114235256700934,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114235414249473} /tidb/cdc/default/default/changefeed/status/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":457114237052387332,"min-table-barrier-ts":457114237052387332,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e537199-dad7-4a9e-873f-624cab54beea/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/841af78d-d649-4521-b501-2fdee5c1fa33/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/5802fee1-da5a-49de-a9a5-fe85cc939ead {UpstreamID:7489359548330505714 Namespace:default ID:5802fee1-da5a-49de-a9a5-fe85cc939ead SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:23.913639834 +0800 CST StartTs:457114235256700934 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba7950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114235414249473} {CheckpointTs:457114237052387332 MinTableBarrierTs:457114237052387332 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e537199-dad7-4a9e-873f-624cab54beea {"id":"4e537199-dad7-4a9e-873f-624cab54beea","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752413} /tidb/cdc/default/__cdc_meta__/capture/841af78d-d649-4521-b501-2fdee5c1fa33 {"id":"841af78d-d649-4521-b501-2fdee5c1fa33","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180acf 4e537199-dad7-4a9e-873f-624cab54beea /tidb/cdc/default/__cdc_meta__/owner/223195ffbe180b77 841af78d-d649-4521-b501-2fdee5c1fa33 /tidb/cdc/default/default/changefeed/info/5802fee1-da5a-49de-a9a5-fe85cc939ead {"upstream-id":7489359548330505714,"namespace":"default","changefeed-id":"5802fee1-da5a-49de-a9a5-fe85cc939ead","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-24277?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:23.913639834+08:00","start-ts":457114235256700934,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457+ grep -q 'etcd info' 114235414249473} /tidb/cdc/default/default/changefeed/status/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":457114237052387332,"min-table-barrier-ts":457114237052387332,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e537199-dad7-4a9e-873f-624cab54beea/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/841af78d-d649-4521-b501-2fdee5c1fa33/5802fee1-da5a-49de-a9a5-fe85cc939ead {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359548330505714 {"id":7489359548330505714,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later + set +x [Fri Apr 4 15:40:30 CST 2025] <<<<<< START Pulsar consumer in batch_add_table case >>>>>> [Fri Apr 4 15:40:30 CST 2025] <<<<<< START Pulsar consumer in capture_session_done_during_task case >>>>>> lease 223195ffbe29f00f revoked TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later table batch_add_table.finish_mark not exists for 1-th check, retry later table capture_session_done_during_task.t exists check diff failed 1-th time, retry later check diff failed 4-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 6-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.9926.out cli tso query --pd=http://127.0.0.1:2379 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:31 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4015 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228412121090,"min-table-barrier-ts":457114228412121090,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228412121090,"min-table-barrier-ts":457114228412121090,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 + grep -q 'failed to get info:' {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489359412972197076 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:39:53.659268132 +0800 CST StartTs:457114227284639749 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0013eeab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114227481247746} {CheckpointTs:457114228267679764 MinTableBarrierTs:457114228267679764 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/74de3f3d-3e3a-46d8-9166-1bb72669b589 {"id":"74de3f3d-3e3a-46d8-9166-1bb72669b589","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752422} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd9fe57a 74de3f3d-3e3a-46d8-9166-1bb72669b589 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489359412972197076,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-8185?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:39:53.659268132+08:00","start-ts":457114227284639749,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114227481247746} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114228412121090,"min-table-barrier-ts":457114228412121090,"admin-job-type":0} /tidb/cdc/default/default/task/position/74de3f3d-3e3a-46d8-9166-1bb72669b589/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359412972197076 + grep -q 'etcd info' {"id":7489359412972197076,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:32 CST 2025] <<<<<< run test case tiflash success! >>>>>> TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + set +x + tso='457114237640900609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114237640900609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "scanproportion"="0" "readallfields"="true" "requestdistribution"="uniform" "mysql.db"="multi_capture_1" "dotransactions"="false" "recordcount"="10" "updateproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "threadcount"="2" "readproportion"="0" "operationcount"="0" "insertproportion"="0" "mysql.user"="root" "mysql.port"="4000" ********************************************** table batch_add_table.finish_mark not exists for 2-th check, retry later Run finished, takes 9.931719ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1931.3, Avg(us): 1884, Min(us): 1028, Max(us): 4669, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "mysql.host"="127.0.0.1" "updateproportion"="0" "readallfields"="true" "mysql.db"="multi_capture_2" "insertproportion"="0" "readproportion"="0" "threadcount"="2" "scanproportion"="0" "requestdistribution"="uniform" "workload"="core" "mysql.user"="root" "operationcount"="0" "dotransactions"="false" "recordcount"="10" "mysql.port"="4000" ********************************************** Run finished, takes 9.070572ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2003.9, Avg(us): 1735, Min(us): 1028, Max(us): 3979, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "recordcount"="10" "readallfields"="true" "threadcount"="2" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "scanproportion"="0" "workload"="core" "updateproportion"="0" "insertproportion"="0" "readproportion"="0" "mysql.user"="root" "mysql.db"="multi_capture_3" "dotransactions"="false" "operationcount"="0" "mysql.port"="4000" ********************************************** Run finished, takes 8.42455ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2156.1, Avg(us): 1607, Min(us): 966, Max(us): 3687, 95th(us): 4000, 99th(us): 4000 table test.finish_mark not exists for 7-th check, retry later check diff failed 5-th time, retry later check diff failed 2-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ***************** properties ***************** "threadcount"="2" "workload"="core" "readallfields"="true" "scanproportion"="0" "readproportion"="0" "recordcount"="10" "mysql.db"="multi_capture_4" "dotransactions"="false" "requestdistribution"="uniform" "updateproportion"="0" "operationcount"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "mysql.port"="4000" ********************************************** Run finished, takes 8.710099ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2039.3, Avg(us): 1662, Min(us): 1044, Max(us): 3763, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:40:34 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.1003910041.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 2-th check, retry later go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/google/uuid v1.6.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/sys v0.28.0 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/net v0.33.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 check data successfully go: downloading golang.org/x/text v0.21.0 public/default ++ date + echo '[Fri Apr 4 15:40:35 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:35 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.3360.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 2829 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later wait process cdc.test exit for 1-th time... table batch_add_table.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... check diff successfully check diff failed 3-th time, retry later Create changefeed successfully! ID: 6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb Info: {"upstream_id":7489359564231334421,"namespace":"default","id":"6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb","sink_uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:36.197964539+08:00","start_ts":457114238459052038,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114238459052038,"checkpoint_ts":457114238459052038,"checkpoint_time":"2025-04-04 15:40:35.528"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:36 CST 2025] <<<<<< run test case simple success! >>>>>> ++ date + echo '[Fri Apr 4 15:40:36 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:36 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6691.out cli changefeed create --start-ts=457114235597750273 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error table ddl_manager.finish_mark not exists for 3-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/gorilla/mux v1.8.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading golang.org/x/time v0.7.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:37 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:40:37 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.1009810100.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 4-th check, retry later Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7489359599843045520,"namespace":"default","id":"changefeed-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:37.534193967+08:00","start_ts":457114235597750273,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114235597750273,"checkpoint_ts":457114235597750273,"checkpoint_time":"2025-04-04 15:40:24.613"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/cockroachdb/redact v1.1.5 + set +x [Fri Apr 4 15:40:37 CST 2025] <<<<<< START Pulsar consumer in owner_resign case >>>>>> cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' table test.finish_mark not exists for 9-th check, retry later check diff successfully check diff failed 1-th time, retry later go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading k8s.io/apimachinery v0.29.11 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b table ddl_manager.finish_mark not exists for 4-th check, retry later + set +x [Fri Apr 4 15:40:38 CST 2025] <<<<<< START Pulsar consumer in changefeed_error case >>>>>> check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:40:37.663453644+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at 457114235597750273" } }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:40:37.663453644+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at 457114235597750273" } }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:40:37.663453644+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at 457114235597750273" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114235597750273, '"checkpoint_time":' '"2025-04-04' '15:40:24.613",' '"error":' '{' '"time":' '"2025-04-04T15:40:37.663453644+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at '457114235597750273"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114235597750273, '"checkpoint_time":' '"2025-04-04' '15:40:24.613",' '"error":' '{' '"time":' '"2025-04-04T15:40:37.663453644+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at '457114235597750273"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at 457114235597750273' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114235597749973 is earlier than or equal to GC safepoint at 457114235597750273 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6824.out cli changefeed resume -c changefeed-error go: downloading github.com/pierrec/lz4 v2.6.1+incompatible table batch_add_table.finish_mark not exists for 5-th check, retry later table owner_resign.t1 exists PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.finish_mark not exists for 10-th check, retry later check diff successfully table ddl_manager.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:40 GMT < Content-Length: 1275 < Content-Type: text/plain; charset=utf-8 < { [1275 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:40:40 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.1015610158.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8303 failed: Connection refused * Failed to connect to 127.0.0.1 port 8303: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 table batch_add_table.finish_mark not exists for 6-th check, retry later wait process cdc.test exit for 2-th time... + set +x table changefeed_error.usertable not exists for 1-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:41 CST 2025] <<<<<< run test case capture_session_done_during_task success! >>>>>> check diff failed 2-th time, retry later table test.finish_mark not exists for 11-th check, retry later table ddl_manager.finish_mark not exists for 6-th check, retry later check diff failed 2-th time, retry later check diff successfully [Fri Apr 4 15:40:42 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.35853587.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table changefeed_error.usertable exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8303 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:43 GMT < Content-Length: 1756 < Content-Type: text/plain; charset=utf-8 < { [1756 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1462d5fe-0221-4e58-894a-bf9b1059cef7 {"id":"1462d5fe-0221-4e58-894a-bf9b1059cef7","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752440} /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627277 1462d5fe-0221-4e58-894a-bf9b1059cef7 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1462d5fe-0221-4e58-894a-bf9b1059cef7 {"id":"1462d5fe-0221-4e58-894a-bf9b1059cef7","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752440} /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627277 1462d5fe-0221-4e58-894a-bf9b1059cef7 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1462d5fe-0221-4e58-894a-bf9b1059cef7 {"id":"1462d5fe-0221-4e58-894a-bf9b1059cef7","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752440} /tidb/cdc/default/__cdc_meta__/capture/49ab227c-c6be-439e-a5ea-fb7279cbf726 {"id":"49ab227c-c6be-439e-a5ea-fb7279cbf726","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752437} /tidb/cdc/default/__cdc_meta__/capture/cfa330f3-5a2a-4aca-a417-f7462da00d69 {"id":"cfa330f3-5a2a-4aca-a417-f7462da00d69","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752434} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627217 cfa330f3-5a2a-4aca-a417-f7462da00d69 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627252 49ab227c-c6be-439e-a5ea-fb7279cbf726 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe627277 1462d5fe-0221-4e58-894a-bf9b1059cef7 /tidb/cdc/default/default/upstream/7489359631824678661 {"id":7489359631824678661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:40:43 CST 2025] <<<<<< START pulsar cluster in multi_capture case >>>>>>' [Fri Apr 4 15:40:43 CST 2025] <<<<<< START pulsar cluster in multi_capture case >>>>>> + workdir=/tmp/tidb_cdc_test/multi_capture + cluster_type=normal + cd /tmp/tidb_cdc_test/multi_capture + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/multi_capture/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/multi_capture/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/multi_capture/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/multi_capture/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/multi_capture/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/multi_capture/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/multi_capture/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/multi_capture/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/multi_capture/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:43 CST 2025] <<<<<< START pulsar cluster in normal mode in multi_capture case >>>>>>' [Fri Apr 4 15:40:43 CST 2025] <<<<<< START pulsar cluster in normal mode in multi_capture case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/multi_capture/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/multi_capture/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/multi_capture/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table batch_add_table.finish_mark not exists for 7-th check, retry later check diff failed 3-th time, retry later table test.finish_mark not exists for 12-th check, retry later table ddl_manager.finish_mark not exists for 7-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:45 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {UpstreamID:7489359564231334421 Namespace:default ID:6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:36.197964539 +0800 CST StartTs:457114238459052038 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f3de60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114238629445635} {CheckpointTs:457114240831455242 MinTableBarrierTs:457114240831455242 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/970f9f59-e13d-41e1-b27e-670f1f167f4b {"id":"970f9f59-e13d-41e1-b27e-670f1f167f4b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2369bd 970f9f59-e13d-41e1-b27e-670f1f167f4b /tidb/cdc/default/default/changefeed/info/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"upstream-id":7489359564231334421,"namespace":"default","changefeed-id":"6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:36.197964539+08:00","start-ts":457114238459052038,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238629445635} /tidb/cdc/default/default/changefeed/status/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":457114240831455242,"min-table-barrier-ts":457114240831455242,"admin-job-type":0} /tidb/cdc/default/default/task/position/970f9f59-e13d-41e1-b27e-670f1f167f4b/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97b17654-e84b-46d7-8072-17548f3666d6/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {UpstreamID:7489359564231334421 Namespace:default ID:6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:36.197964539 +0800 CST StartTs:457114238459052038 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f3de60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114238629445635} {CheckpointTs:457114240831455242 MinTableBarrierTs:457114240831455242 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/970f9f59-e13d-41e1-b27e-670f1f167f4b {"id":"970f9f59-e13d-41e1-b27e-670f1f167f4b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2369bd 970f9f59-e13d-41e1-b27e-670f1f167f4b /tidb/cdc/default/default/changefeed/info/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"upstream-id":7489359564231334421,"namespace":"default","changefeed-id":"6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:36.197964539+08:00","start-ts":457114238459052038,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238629445635} /tidb/cdc/default/default/changefeed/status/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":457114240831455242,"min-table-barrier-ts":457114240831455242,"admin-job-type":0} /tidb/cdc/default/default/task/position/970f9f59-e13d-41e1-b27e-670f1f167f4b/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97b17654-e84b-46d7-8072-17548f3666d6/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {UpstreamID:7489359564231334421 Namespace:default ID:6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:40:36.197964539 +0800 CST StartTs:457114238459052038 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f3de60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114238629445635} {CheckpointTs:457114240831455242 MinTableBarrierTs:457114240831455242 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/970f9f59-e13d-41e1-b27e-670f1f167f4b {"id":"970f9f59-e13d-41e1-b27e-670f1f167f4b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/capture/97b17654-e84b-46d7-8072-17548f3666d6 {"id":"97b17654-e84b-46d7-8072-17548f3666d6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752425} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2368ef 97b17654-e84b-46d7-8072-17548f3666d6 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe2369bd 970f9f59-e13d-41e1-b27e-670f1f167f4b /tidb/cdc/default/default/changefeed/info/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"upstream-id":7489359564231334421,"namespace":"default","changefeed-id":"6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-610?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:36.197964539+08:00","start-ts":457114238459052038,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114238629445635} /tidb/cdc/default/default/changefeed/status/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":457114240831455242,"min-table-barrier-ts":457114240831455242,"admin-job-type":0} /tidb/cdc/default/default/task/position/970f9f59-e13d-41e1-b27e-670f1f167f4b/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97b17654-e84b-46d7-8072-17548f3666d6/6a4c7015-f86d-448a-b0ff-74bc9a1bcbfb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564231334421 {"id":7489359564231334421,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 table batch_add_table.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... check diff successfully ***************** properties ***************** "readallfields"="true" "mysql.db"="changefeed_error" "updateproportion"="0" "mysql.port"="4000" "dotransactions"="false" "mysql.host"="127.0.0.1" "mysql.user"="root" "insertproportion"="0" "recordcount"="20" "scanproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "workload"="core" "threadcount"="4" "readproportion"="0" ********************************************** Run finished, takes 4.655399ms INSERT - Takes(s): 0.0, Count: 20, OPS: 7020.7, Avg(us): 835, Min(us): 480, Max(us): 2053, 95th(us): 2000, 99th(us): 3000 check diff successfully {"id":"b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752427} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + [[ /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore table test.finish_mark exists Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:46 CST 2025] <<<<<< run test case split_region success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff successfully check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 8-th check, retry later table batch_add_table.finish_mark not exists for 9-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + [[ /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359712056656439 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb18e0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:7965, start at 2025-04-04 15:40:47.563150667 +0800 CST m=+1.546885617 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.570 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359712056656439 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb18e0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:7965, start at 2025-04-04 15:40:47.563150667 +0800 CST m=+1.546885617 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.570 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359709241893500 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb19a8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:8033, start at 2025-04-04 15:40:47.631716569 +0800 CST m=+1.543771195 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.640 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.644 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.644 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 9-th check, retry later table batch_add_table.finish_mark not exists for 10-th check, retry later check diff successfully public/default ++ date + echo '[Fri Apr 4 15:40:49 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:49 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.10633.out cli changefeed create --start-ts=457114237640900609 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-multi-capture-test-24165?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 10-th check, retry later check diff failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Create changefeed successfully! ID: 99adafa4-1f62-4510-a5ad-bef652b57914 Info: {"upstream_id":7489359631824678661,"namespace":"default","id":"99adafa4-1f62-4510-a5ad-bef652b57914","sink_uri":"pulsar://127.0.0.1:6650/ticdc-multi-capture-test-24165?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:40:50.704195401+08:00","start_ts":457114237640900609,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114237640900609,"checkpoint_ts":457114237640900609,"checkpoint_time":"2025-04-04 15:40:32.407"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ++ date + echo '[Fri Apr 4 15:40:50 CST 2025] <<<<<< START pulsar cluster in processor_stop_delay case >>>>>>' [Fri Apr 4 15:40:50 CST 2025] <<<<<< START pulsar cluster in processor_stop_delay case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_stop_delay + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_stop_delay + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_stop_delay/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_stop_delay/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_stop_delay/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_stop_delay/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_stop_delay/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_stop_delay/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_stop_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_stop_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_stop_delay/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:40:50 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_stop_delay case >>>>>>' [Fri Apr 4 15:40:50 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_stop_delay case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_stop_delay/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_stop_delay/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_stop_delay/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff successfully wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark not exists for 11-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:51 CST 2025] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + [[ /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b9e3666e-4ba1-49a5-bde7-2d6c552e2f0b + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + set +x [Fri Apr 4 15:40:52 CST 2025] <<<<<< START Pulsar consumer in multi_capture case >>>>>> table multi_capture_1.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 11-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 12-th check, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359748919239648 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb8fdc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:4778, start at 2025-04-04 15:40:55.178354008 +0800 CST m=+1.567106025 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:55.186 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:55.159 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:55.159 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359748919239648 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb8fdc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:4778, start at 2025-04-04 15:40:55.178354008 +0800 CST m=+1.567106025 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:55.186 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:55.159 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:55.159 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359741447934740 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb90900010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-wb2g-49z6s, pid:4859, start at 2025-04-04 15:40:55.240389002 +0800 CST m=+1.556256239 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:55.251 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:55.254 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:55.254 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table batch_add_table.finish_mark not exists for 13-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) table test.finish_mark not exists for 1-th check, retry later [Pipeline] // cache [Pipeline] } check diff failed 2-th time, retry later [Pipeline] // dir public/default table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table ddl_manager.finish_mark not exists for 13-th check, retry later [Pipeline] // node [Pipeline] } ++ date + echo '[Fri Apr 4 15:40:56 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:40:56 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:40:56 CST 2025] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.98379839.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table batch_add_table.finish_mark not exists for 14-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746353078645 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbaa3c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:4623, start at 2025-04-04 15:40:56.86279791 +0800 CST m=+1.546953611 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:56.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:56.847 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:56.847 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359746353078645 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbaa3c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:4623, start at 2025-04-04 15:40:56.86279791 +0800 CST m=+1.546953611 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:56.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:56.847 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:56.847 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359757606928104 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbab10000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:4704, start at 2025-04-04 15:40:56.942286022 +0800 CST m=+1.550104633 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:56.951 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:56.950 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:56.950 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 2-th check, retry later table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:40:57 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.70737075.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Fri Apr 4 15:40:58 CST 2025] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.62316233.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 14-th check, retry later check diff failed 3-th time, retry later table batch_add_table.finish_mark not exists for 15-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.6171.out cli tso query --pd=http://127.0.0.1:2379 table test.finish_mark not exists for 3-th check, retry later table multi_capture_4.usertable exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:59 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2063260-a2ca-4d80-b923-1af7f660d160 {"id":"d2063260-a2ca-4d80-b923-1af7f660d160","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbeaa44f3 d2063260-a2ca-4d80-b923-1af7f660d160 /tidb/cdc/default/default/upstream/7489359712056656439 {"id":7489359712056656439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2063260-a2ca-4d80-b923-1af7f660d160 {"id":"d2063260-a2ca-4d80-b923-1af7f660d160","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbeaa44f3 d2063260-a2ca-4d80-b923-1af7f660d160 /tidb/cdc/default/default/upstream/7489359712056656439 {"id":7489359712056656439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2063260-a2ca-4d80-b923-1af7f660d160 {"id":"d2063260-a2ca-4d80-b923-1af7f660d160","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752457} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbeaa44f3 d2063260-a2ca-4d80-b923-1af7f660d160 /tidb/cdc/default/default/upstream/7489359712056656439 {"id":7489359712056656439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 15-th check, retry later check diff failed 4-th time, retry later [Fri Apr 4 15:41:00 CST 2025] <<<<<< START Pulsar consumer in processor_stop_delay case >>>>>> table processor_stop_delay.t not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3521 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/214e1a4e-aba3-485d-af8a-5264ba5ff2c7 {"id":"214e1a4e-aba3-485d-af8a-5264ba5ff2c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f26 214e1a4e-aba3-485d-af8a-5264ba5ff2c7 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489359599843045520,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:37.534193967+08:00","start-ts":457114235597750273,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:40:58.392424963+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114244444848132} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114240919273479,"min-table-barrier-ts":457114240919273479,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/214e1a4e-aba3-485d-af8a-5264ba5ff2c7 {"id":"214e1a4e-aba3-485d-af8a-5264ba5ff2c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f26 214e1a4e-aba3-485d-af8a-5264ba5ff2c7 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489359599843045520,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:37.534193967+08:00","start-ts":457114235597750273,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:40:58.392424963+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114244444848132} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114240919273479,"min-table-barrier-ts":457114240919273479,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/214e1a4e-aba3-485d-af8a-5264ba5ff2c7 {"id":"214e1a4e-aba3-485d-af8a-5264ba5ff2c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f26 214e1a4e-aba3-485d-af8a-5264ba5ff2c7 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489359599843045520,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:40:37.534193967+08:00","start-ts":457114235597750273,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:40:58.392424963+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114244444848132} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114240919273479,"min-table-barrier-ts":457114240919273479,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114240919273479, "checkpoint_time": "2025-04-04 15:40:44.913", "error": { "time": "2025-04-04T15:40:58.392424963+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114240919273479, "checkpoint_time": "2025-04-04 15:40:44.913", "error": { "time": "2025-04-04T15:40:58.392424963+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114240919273479, "checkpoint_time": "2025-04-04 15:40:44.913", "error": { "time": "2025-04-04T15:40:58.392424963+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114240919273479, '"checkpoint_time":' '"2025-04-04' '15:40:44.913",' '"error":' '{' '"time":' '"2025-04-04T15:40:58.392424963+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114240919273479, '"checkpoint_time":' '"2025-04-04' '15:40:44.913",' '"error":' '{' '"time":' '"2025-04-04T15:40:58.392424963+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7168.out cli changefeed remove -c changefeed-error < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:01 GMT < Content-Length: 861 < Content-Type: text/plain; charset=utf-8 < { [861 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c6abb84-2851-4666-9b1e-a204ffdd97fc {"id":"2c6abb84-2851-4666-9b1e-a204ffdd97fc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec9c12e 2c6abb84-2851-4666-9b1e-a204ffdd97fc /tidb/cdc/default/default/upstream/7489359748919239648 {"id":7489359748919239648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c6abb84-2851-4666-9b1e-a204ffdd97fc {"id":"2c6abb84-2851-4666-9b1e-a204ffdd97fc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec9c12e 2c6abb84-2851-4666-9b1e-a204ffdd97fc /tidb/cdc/default/default/upstream/7489359748919239648 {"id":7489359748919239648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c6abb84-2851-4666-9b1e-a204ffdd97fc {"id":"2c6abb84-2851-4666-9b1e-a204ffdd97fc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752458} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec9c12e 2c6abb84-2851-4666-9b1e-a204ffdd97fc /tidb/cdc/default/default/upstream/7489359748919239648 {"id":7489359748919239648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:41:01 CST 2025] <<<<<< START pulsar cluster in region_merge case >>>>>>' [Fri Apr 4 15:41:01 CST 2025] <<<<<< START pulsar cluster in region_merge case >>>>>> + workdir=/tmp/tidb_cdc_test/region_merge + cluster_type=mtls + cd /tmp/tidb_cdc_test/region_merge + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/region_merge/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/region_merge/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/region_merge/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/region_merge/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/region_merge/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ............................+++++ .......................+++++ e is 65537 (0x010001) + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ..............................................................+++++ \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } ............................................................................................+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=CN = server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ......................+++++ ......+++++ e is 65537 (0x010001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=CN = broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/region_merge/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/region_merge/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem' ++ date + echo '[Fri Apr 4 15:41:01 CST 2025] <<<<<< START pulsar cluster in mtls mode in region_merge case >>>>>>' [Fri Apr 4 15:41:01 CST 2025] <<<<<< START pulsar cluster in mtls mode in region_merge case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/region_merge/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/region_merge/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/region_merge/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir + set +x + tso='457114244886560769 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114244886560769 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Fri Apr 4 15:41:01 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.62126214.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } Changefeed remove successfully. ID: changefeed-error CheckpointTs: 457114240919273479 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true PASS [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } coverage: 2.0% of statements in github.com/pingcap/tiflow/... table batch_add_table.finish_mark not exists for 16-th check, retry later [Pipeline] // withEnv table test.finish_mark not exists for 4-th check, retry later [Pipeline] } [Pipeline] // node VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359774201195496 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbf4c0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:4817, start at 2025-04-04 15:41:01.653598129 +0800 CST m=+1.642151619 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:01.662 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:01.666 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:01.666 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359774201195496 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbf4c0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:4817, start at 2025-04-04 15:41:01.653598129 +0800 CST m=+1.642151619 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:01.662 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:01.666 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:01.666 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359770674411781 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbf59c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-cf30-nblpp, pid:4902, start at 2025-04-04 15:41:01.713020787 +0800 CST m=+1.623688604 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:01.720 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:01.721 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:01.721 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] } table ddl_manager.finish_mark not exists for 16-th check, retry later check diff successfully check diff successfully ***************** properties ***************** "mysql.user"="root" "readallfields"="true" "threadcount"="2" "mysql.db"="multi_capture_1" "workload"="core" "readproportion"="0" "updateproportion"="0" "requestdistribution"="uniform" "recordcount"="20" "mysql.port"="4000" "mysql.host"="127.0.0.1" "insertproportion"="0" "dotransactions"="false" "scanproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 10.769893ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2139.4, Avg(us): 1021, Min(us): 469, Max(us): 1773, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "threadcount"="2" "updateproportion"="0" "insertproportion"="0" "scanproportion"="0" "readproportion"="0" "dotransactions"="false" "mysql.port"="4000" "operationcount"="0" "readallfields"="true" "workload"="core" "mysql.host"="127.0.0.1" "recordcount"="20" "mysql.user"="root" "requestdistribution"="uniform" "mysql.db"="multi_capture_2" ********************************************** Run finished, takes 9.53126ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2362.1, Avg(us): 897, Min(us): 433, Max(us): 1546, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "mysql.port"="4000" "requestdistribution"="uniform" "readproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "dotransactions"="false" "recordcount"="20" "mysql.db"="multi_capture_3" "scanproportion"="0" "threadcount"="2" "mysql.user"="root" "updateproportion"="0" "readallfields"="true" "insertproportion"="0" "workload"="core" ********************************************** Run finished, takes 9.86014ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2263.7, Avg(us): 939, Min(us): 443, Max(us): 1554, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "mysql.db"="multi_capture_4" "threadcount"="2" "updateproportion"="0" "recordcount"="20" "dotransactions"="false" "operationcount"="0" "mysql.port"="4000" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.user"="root" "workload"="core" ********************************************** Run finished, takes 11.049976ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1995.8, Avg(us): 1043, Min(us): 452, Max(us): 1812, 95th(us): 2000, 99th(us): 2000 check diff failed 1-th time, retry later check diff successfully + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully + nc -z localhost 6651 table processor_stop_delay.t exists check diff failed 1-th time, retry later + i=2 + '[' 2 -gt 20 ']' + sleep 2 wait process cdc.test exit for 1-th time... table batch_add_table.finish_mark not exists for 17-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.6347.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:04 CST 2025] <<<<<< run test case owner_resign success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:04 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d496c90-bc76-4d9e-a310-50092f2f7ee1 {"id":"7d496c90-bc76-4d9e-a310-50092f2f7ee1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752461} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed05ce3 7d496c90-bc76-4d9e-a310-50092f2f7ee1 /tidb/cdc/default/default/upstream/7489359746353078645 {"id":7489359746353078645,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d496c90-bc76-4d9e-a310-50092f2f7ee1 {"id":"7d496c90-bc76-4d9e-a310-50092f2f7ee1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752461} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed05ce3 7d496c90-bc76-4d9e-a310-50092f2f7ee1 /tidb/cdc/default/default/upstream/7489359746353078645 {"id":7489359746353078645,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d496c90-bc76-4d9e-a310-50092f2f7ee1 {"id":"7d496c90-bc76-4d9e-a310-50092f2f7ee1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752461} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed05ce3 7d496c90-bc76-4d9e-a310-50092f2f7ee1 /tidb/cdc/default/default/upstream/7489359746353078645 {"id":7489359746353078645,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:41:04 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.72457247.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 17-th check, retry later check diff successfully check diff successfully + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public wait process cdc.test exit for 1-th time... table batch_add_table.finish_mark exists check diff successfully wait process cdc.test exit for 2-th time... + set +x + tso='457114246110511105 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114246110511105 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:06 CST 2025] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.63856387.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... table test.finish_mark exists check diff successfully process cdc.test already exit [Fri Apr 4 15:41:06 CST 2025] <<<<<< run test case multi_capture success! >>>>>> wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 18-th check, retry later check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:07 CST 2025] <<<<<< run test case canal_json_content_compatible success! >>>>>> public/default ++ date + echo '[Fri Apr 4 15:41:08 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:41:08 CST 2025] <<<<<< pulsar is ready >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:07 GMT < Content-Length: 842 < Content-Type: text/plain; charset=utf-8 < { [842 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afe60224-4c65-41d3-83d2-fa8fdf56b708 {"id":"afe60224-4c65-41d3-83d2-fa8fdf56b708","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752465} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f7e afe60224-4c65-41d3-83d2-fa8fdf56b708 /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afe60224-4c65-41d3-83d2-fa8fdf56b708 {"id":"afe60224-4c65-41d3-83d2-fa8fdf56b708","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752465} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f7e afe60224-4c65-41d3-83d2-fa8fdf56b708 /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afe60224-4c65-41d3-83d2-fa8fdf56b708 {"id":"afe60224-4c65-41d3-83d2-fa8fdf56b708","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752465} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0f7e afe60224-4c65-41d3-83d2-fa8fdf56b708 /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7297.out cli changefeed create --start-ts=457114235597750273 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7489359599843045520,"namespace":"default","id":"changefeed-error-1","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:08.424832614+08:00","start_ts":457114235597750273,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114235597750273,"checkpoint_ts":457114235597750273,"checkpoint_time":"2025-04-04 15:40:24.613"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later Create changefeed successfully! ID: 6c1bc679-4950-4e63-8ac4-78284a511d84 Info: {"upstream_id":7489359748919239648,"namespace":"default","id":"6c1bc679-4950-4e63-8ac4-78284a511d84","sink_uri":"pulsar+ssl://127.0.0.1:6651/ticdc-region-merge-test-24608?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:08.877756099+08:00","start_ts":457114247039287298,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"tls-trust-certs-file-path":"/tmp/tidb_cdc_test/region_merge/ca.cert.pem","connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"auth-tls-certificate-path":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","auth-tls-private-key-path":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114247039287298,"checkpoint_ts":457114247039287298,"checkpoint_time":"2025-04-04 15:41:08.259"} [Fri Apr 4 15:41:08 CST 2025] <<<<<< START Pulsar consumer in region_merge case >>>>>> table ddl_manager.finish_mark not exists for 19-th check, retry later check diff failed 2-th time, retry later split_and_random_merge scale: 20 *************************** 1. row *************************** count(distinct region_id): 1 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:09 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95771aea-05bb-48fc-814b-d51b201314a0 {"id":"95771aea-05bb-48fc-814b-d51b201314a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbee2cad8 95771aea-05bb-48fc-814b-d51b201314a0 /tidb/cdc/default/default/upstream/7489359774201195496 {"id":7489359774201195496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95771aea-05bb-48fc-814b-d51b201314a0 {"id":"95771aea-05bb-48fc-814b-d51b201314a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbee2cad8 95771aea-05bb-48fc-814b-d51b201314a0 /tidb/cdc/default/default/upstream/7489359774201195496 {"id":7489359774201195496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95771aea-05bb-48fc-814b-d51b201314a0 {"id":"95771aea-05bb-48fc-814b-d51b201314a0","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbee2cad8 95771aea-05bb-48fc-814b-d51b201314a0 /tidb/cdc/default/default/upstream/7489359774201195496 {"id":7489359774201195496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:41:09 CST 2025] <<<<<< START pulsar cluster in ddl_attributes case >>>>>>' [Fri Apr 4 15:41:09 CST 2025] <<<<<< START pulsar cluster in ddl_attributes case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_attributes + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_attributes + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_attributes/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_attributes/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_attributes/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_attributes/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_attributes/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_attributes/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_attributes/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_attributes/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_attributes/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:41:09 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_attributes case >>>>>>' [Fri Apr 4 15:41:09 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_attributes case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_attributes/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_attributes/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_attributes/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 3162 0 --:--:-- --:--:-- --:--:-- 3162 + info='{"state":"warning","resolved_ts":457114235977596946,"checkpoint_ts":457114235977596946,"last_warning":{"time":"2025-04-04T15:41:10.198981009+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":457114235977596946,"checkpoint_ts":457114235977596946,"last_warning":{"time":"2025-04-04T15:41:10.198981009+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":457114235977596946,"checkpoint_ts":457114235977596946,"last_warning":{"time":"2025-04-04T15:41:10.198981009+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":457114235977596946,"checkpoint_ts":457114235977596946,"last_warning":{"time":"2025-04-04T15:41:10.198981009+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":457114235977596946,"checkpoint_ts":457114235977596946,"last_warning":{"time":"2025-04-04T15:41:10.198981009+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7344.out cli changefeed remove -c changefeed-error-1 table ddl_manager.finish_mark not exists for 20-th check, retry later check diff failed 3-th time, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 457114235977596946 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true PASS + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff failed 3-th time, retry later coverage: 2.0% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 21-th check, retry later + set +x check diff failed 4-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public check diff failed 4-th time, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:41:14 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.74027404.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later check diff failed 5-th time, retry later *************************** 1. row *************************** count(distinct region_id): 20 public/default ++ date + echo '[Fri Apr 4 15:41:15 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:41:15 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.6850.out cli changefeed create --start-ts=457114246110511105 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-20625?protocol=canal-json&enable-tidb-extension=true' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... Create changefeed successfully! ID: cd3ce542-5982-4974-8a90-b38eb2c6e1c9 Info: {"upstream_id":7489359774201195496,"namespace":"default","id":"cd3ce542-5982-4974-8a90-b38eb2c6e1c9","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-20625?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:16.55069511+08:00","start_ts":457114246110511105,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114246110511105,"checkpoint_ts":457114246110511105,"checkpoint_time":"2025-04-04 15:41:04.716"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 23-th check, retry later shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory check diff failed 6-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:17 GMT < Content-Length: 842 < Content-Type: text/plain; charset=utf-8 < { [842 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d {"id":"ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0ff8 ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d {"id":"ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0ff8 ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d {"id":"ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a0ff8 ff42f2f6-1a80-4a11-ae44-b9ad1e405d1d /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7456.out cli changefeed create --start-ts=457114235597750273 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:17 CST 2025] <<<<<< run test case multi_topics success! >>>>>> check diff failed 6-th time, retry later Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7489359599843045520,"namespace":"default","id":"changefeed-error-2","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:17.939882249+08:00","start_ts":457114235597750273,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114235597750273,"checkpoint_ts":457114235597750273,"checkpoint_time":"2025-04-04 15:40:24.613"} PASS + set +x [Fri Apr 4 15:41:17 CST 2025] <<<<<< START Pulsar consumer in ddl_attributes case >>>>>> coverage: 2.5% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 24-th check, retry later check diff successfully + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:41:18.017385425+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:41:18.017385425+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114235597750273, "checkpoint_time": "2025-04-04 15:40:24.613", "error": { "time": "2025-04-04T15:41:18.017385425+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114235597750273, '"checkpoint_time":' '"2025-04-04' '15:40:24.613",' '"error":' '{' '"time":' '"2025-04-04T15:41:18.017385425+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114235597750273, '"checkpoint_time":' '"2025-04-04' '15:40:24.613",' '"error":' '{' '"time":' '"2025-04-04T15:41:18.017385425+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114235597750273 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7515.out cli changefeed remove -c changefeed-error-2 check diff successfully wait process cdc.test exit for 1-th time... Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 457114235597750273 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 25-th check, retry later split_and_random_merge scale: 40 *************************** 1. row *************************** count(distinct region_id): 9 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:21 CST 2025] <<<<<< run test case processor_stop_delay success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:20 CST 2025] <<<<<< run test case avro_basic success! >>>>>> + set +x table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 26-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... Verifying downstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:23 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' check diff failed 1-th time, retry later + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.75727574.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:24 CST 2025] <<<<<< run test case canal_json_handle_key_only success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 27-th check, retry later check diff failed 2-th time, retry later table ddl_attributes.attributes_t1_new not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359878628238925 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd706c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:4952, start at 2025-04-04 15:41:25.927174475 +0800 CST m=+1.728797946 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.935 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.915 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.915 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359878628238925 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd706c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:4952, start at 2025-04-04 15:41:25.927174475 +0800 CST m=+1.728797946 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.935 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.915 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.915 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359881247769041 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd7028000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:5018, start at 2025-04-04 15:41:25.946327839 +0800 CST m=+1.682703053 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.954 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.948 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.948 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 4 15:41:24 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.63116313.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None + [[ true != \n\o ]] + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:26 GMT < Content-Length: 842 < Content-Type: text/plain; charset=utf-8 < { [842 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0cf323a-6032-4c4c-be68-5be1fc91de4e {"id":"a0cf323a-6032-4c4c-be68-5be1fc91de4e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a107a a0cf323a-6032-4c4c-be68-5be1fc91de4e /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0cf323a-6032-4c4c-be68-5be1fc91de4e {"id":"a0cf323a-6032-4c4c-be68-5be1fc91de4e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a107a a0cf323a-6032-4c4c-be68-5be1fc91de4e /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0cf323a-6032-4c4c-be68-5be1fc91de4e {"id":"a0cf323a-6032-4c4c-be68-5be1fc91de4e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4a107a a0cf323a-6032-4c4c-be68-5be1fc91de4e /tidb/cdc/default/default/upstream/7489359599843045520 {"id":7489359599843045520,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7629.out cli changefeed create --start-ts=0 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7489359599843045520,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:26.868596298+08:00","start_ts":457114251889999876,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114251889999876,"checkpoint_ts":457114251889999876,"checkpoint_time":"2025-04-04 15:41:26.763"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 28-th check, retry later *************************** 1. row *************************** count(distinct region_id): 40 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:27 CST 2025] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> check diff failed 3-th time, retry later table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359878879361867 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd866c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:12068, start at 2025-04-04 15:41:27.365529494 +0800 CST m=+1.580156821 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.373 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359878879361867 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd866c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:12068, start at 2025-04-04 15:41:27.365529494 +0800 CST m=+1.580156821 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.373 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359883648863294 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd8728000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:12134, start at 2025-04-04 15:41:27.396028324 +0800 CST m=+1.537698767 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.404 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.370 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.370 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359886540489538 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd8b48000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:4813, start at 2025-04-04 15:41:27.676111569 +0800 CST m=+1.550098378 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.684 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.683 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.683 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359886540489538 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd8b48000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:4813, start at 2025-04-04 15:41:27.676111569 +0800 CST m=+1.550098378 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.684 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.683 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.683 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359886049827009 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd8da4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-6cb1-2dwc6, pid:4892, start at 2025-04-04 15:41:27.804934036 +0800 CST m=+1.603251465 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:27.815 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:27.785 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:27.785 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ++ date + echo '[Fri Apr 4 15:41:28 CST 2025] <<<<<< START pulsar cluster in processor_etcd_worker_delay case >>>>>>' [Fri Apr 4 15:41:28 CST 2025] <<<<<< START pulsar cluster in processor_etcd_worker_delay case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_etcd_worker_delay + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:41:28 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_etcd_worker_delay case >>>>>>' [Fri Apr 4 15:41:28 CST 2025] <<<<<< START pulsar cluster in normal mode in processor_etcd_worker_delay case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114251889999876, "checkpoint_time": "2025-04-04 15:41:26.763", "error": { "time": "2025-04-04T15:41:27.045110603+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114251889999876, "checkpoint_time": "2025-04-04 15:41:26.763", "error": { "time": "2025-04-04T15:41:27.045110603+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114251889999876, "checkpoint_time": "2025-04-04 15:41:26.763", "error": { "time": "2025-04-04T15:41:27.045110603+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114251889999876, '"checkpoint_time":' '"2025-04-04' '15:41:26.763",' '"error":' '{' '"time":' '"2025-04-04T15:41:27.045110603+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 29-th check, retry later check diff failed 4-th time, retry later table ddl_attributes.finish_mark not exists for 2-th check, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13510.out cli tso query --pd=http://127.0.0.1:2379 + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null } ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7734.out cli changefeed pause -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 30-th check, retry later [Fri Apr 4 15:41:30 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.62276229.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... split_and_random_merge scale: 80 *************************** 1. row *************************** count(distinct region_id): 5 check diff failed 5-th time, retry later + set +x + tso='457114252875661313 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114252875661313 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114252875661313 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:41:31 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1355813560.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:31 CST 2025] <<<<<< run test case canal_json_claim_check success! >>>>>> table ddl_attributes.finish_mark not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 31-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null } ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7792.out cli changefeed resume -c changefeed-initialize-error + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:33 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f16cdd7-5c90-4ad2-b346-5410cba68ec5 {"id":"3f16cdd7-5c90-4ad2-b346-5410cba68ec5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf46ddcf 3f16cdd7-5c90-4ad2-b346-5410cba68ec5 /tidb/cdc/default/default/upstream/7489359886540489538 {"id":7489359886540489538,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f16cdd7-5c90-4ad2-b346-5410cba68ec5 {"id":"3f16cdd7-5c90-4ad2-b346-5410cba68ec5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf46ddcf 3f16cdd7-5c90-4ad2-b346-5410cba68ec5 /tidb/cdc/default/default/upstream/7489359886540489538 {"id":7489359886540489538,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f16cdd7-5c90-4ad2-b346-5410cba68ec5 {"id":"3f16cdd7-5c90-4ad2-b346-5410cba68ec5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf46ddcf 3f16cdd7-5c90-4ad2-b346-5410cba68ec5 /tidb/cdc/default/default/upstream/7489359886540489538 {"id":7489359886540489538,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:41:33 CST 2025] <<<<<< START pulsar cluster in batch_add_table case >>>>>>' [Fri Apr 4 15:41:33 CST 2025] <<<<<< START pulsar cluster in batch_add_table case >>>>>> + workdir=/tmp/tidb_cdc_test/batch_add_table + cluster_type=normal + cd /tmp/tidb_cdc_test/batch_add_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/batch_add_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/batch_add_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:41:33 CST 2025] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>>' [Fri Apr 4 15:41:33 CST 2025] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table ddl_attributes.finish_mark not exists for 4-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:41:35 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:41:35 CST 2025] <<<<<< pulsar is ready >>>>>> [Fri Apr 4 15:41:35 CST 2025] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' + echo '{ "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null }' { "upstream_id": 7489359599843045520, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114252728598534, "checkpoint_time": "2025-04-04 15:41:29.962", "error": null } ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359599843045520, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114252728598534, '"checkpoint_time":' '"2025-04-04' '15:41:29.962",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7861.out cli changefeed remove -c changefeed-initialize-error + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.68066808.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:35 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 {"id":"f5a88f23-4fed-4405-831d-2ec7a4f3cdd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752492} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf4576d8 f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 /tidb/cdc/default/default/upstream/7489359878879361867 {"id":7489359878879361867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 {"id":"f5a88f23-4fed-4405-831d-2ec7a4f3cdd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752492} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf4576d8 f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 /tidb/cdc/default/default/upstream/7489359878879361867 {"id":7489359878879361867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 {"id":"f5a88f23-4fed-4405-831d-2ec7a4f3cdd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752492} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf4576d8 f5a88f23-4fed-4405-831d-2ec7a4f3cdd8 /tidb/cdc/default/default/upstream/7489359878879361867 {"id":7489359878879361867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114252875661313 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13613.out cli changefeed create --start-ts=457114252875661313 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml table ddl_manager.finish_mark not exists for 32-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:34 CST 2025] <<<<<< run test case open_protocol_claim_check success! >>>>>> Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489359878879361867,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:35.455447609+08:00","start_ts":457114252875661313,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114252875661313,"checkpoint_ts":457114252875661313,"checkpoint_time":"2025-04-04 15:41:30.523"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 457114254039580678 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-11909?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table ddl_attributes.finish_mark not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 5139 0 --:--:-- --:--:-- --:--:-- 5139 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:41:30.523","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:41:36.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:41:30.523","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:41:36.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:41:30.523","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:41:36.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:41:30.523' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:41:30.523","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:41:36.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:41:30.523","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:41:36.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:41:36' + echo 'sink_checkpoint_ts is 2025-04-04' 15:41:30.523 sink_checkpoint_ts is 2025-04-04 15:41:30.523 ++ date -d '2025-04-04 15:41:30.523' +%s + checkpoint_timestamp=1743752490 ++ date -d '2025-04-04 15:41:36' +%s + current_timestamp=1743752496 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 33-th check, retry later + set +x wait process cdc.test exit for 1-th time... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table ddl_attributes.finish_mark not exists for 6-th check, retry later *************************** 1. row *************************** count(distinct region_id): 75 check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:38 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/218616b3-a88f-46ec-984b-e6b970c6c051 {"id":"218616b3-a88f-46ec-984b-e6b970c6c051","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752495} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3d5bf6 218616b3-a88f-46ec-984b-e6b970c6c051 /tidb/cdc/default/default/upstream/7489359878628238925 {"id":7489359878628238925,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/218616b3-a88f-46ec-984b-e6b970c6c051 {"id":"218616b3-a88f-46ec-984b-e6b970c6c051","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752495} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3d5bf6 218616b3-a88f-46ec-984b-e6b970c6c051 /tidb/cdc/default/default/upstream/7489359878628238925 {"id":7489359878628238925,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/218616b3-a88f-46ec-984b-e6b970c6c051 {"id":"218616b3-a88f-46ec-984b-e6b970c6c051","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752495} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3d5bf6 218616b3-a88f-46ec-984b-e6b970c6c051 /tidb/cdc/default/default/upstream/7489359878628238925 {"id":7489359878628238925,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:38 CST 2025] <<<<<< run test case canal_json_storage_basic success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:38 CST 2025] <<<<<< run test case changefeed_error success! >>>>>> table ddl_manager.finish_mark not exists for 34-th check, retry later [Fri Apr 4 15:41:38 CST 2025] <<<<<< START Pulsar consumer in processor_etcd_worker_delay case >>>>>> table test.t1 exists + sleep 5 check diff failed 2-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release public/default table ddl_attributes.finish_mark not exists for 7-th check, retry later ++ date + echo '[Fri Apr 4 15:41:40 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:41:40 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.6690.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-5699?protocol=canal-json&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/batch_add_table/conf/changefeed.toml table ddl_manager.finish_mark not exists for 35-th check, retry later Create changefeed successfully! ID: 99757eae-1b5c-4dd8-8f92-bad8240ed665 Info: {"upstream_id":7489359886540489538,"namespace":"default","id":"99757eae-1b5c-4dd8-8f92-bad8240ed665","sink_uri":"pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-5699?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:40.95785963+08:00","start_ts":457114255460663300,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!test.t_1"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114255460663300,"checkpoint_ts":457114255460663300,"checkpoint_time":"2025-04-04 15:41:40.384"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:41:41 CST 2025] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> table ddl_attributes.finish_mark not exists for 8-th check, retry later check diff failed 3-th time, retry later + set +x [Fri Apr 4 15:41:42 CST 2025] <<<<<< START Pulsar consumer in batch_add_table case >>>>>> table test.t_1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 36-th check, retry later table region_merge.t1 exists check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359953913382495 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe79ec000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:11445, start at 2025-04-04 15:41:42.921154485 +0800 CST m=+1.633723691 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.930 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.907 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.907 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359953913382495 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe79ec000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:11445, start at 2025-04-04 15:41:42.921154485 +0800 CST m=+1.633723691 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.930 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.907 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.907 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359952661057002 TiDB Cluster ID. tikv_gc_leader_uuid 657fefe7ab0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k8vq-mtds4, pid:11519, start at 2025-04-04 15:41:42.989323415 +0800 CST m=+1.630921230 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:42.997 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:42.956 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:42.956 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later table ddl_attributes.finish_mark not exists for 9-th check, retry later table test.t_1 exists table test.t_2 exists table test.t_3 exists table test.t_4 exists table test.t_5 exists table test.t_6 exists table test.t_7 exists table test.t_8 exists table test.t_9 exists table test.t_10 exists table test.t_11 exists table test.t_12 exists table test.t_13 exists table test.t_14 exists table test.t_15 exists table test.t_16 exists table test.t_17 exists table test.t_18 exists table test.t_19 exists table test.t_20 exists table test.t_21 exists table test.t_22 exists table test.t_23 exists table test.t_24 exists table test.t_25 exists table test.t_26 exists table test.t_27 exists table test.t_28 exists table test.t_29 exists table test.t_30 exists table test.t_31 exists table test.t_32 exists table test.t_33 exists table test.t_34 exists table test.t_35 exists invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 4 15:41:44 CST 2025] <<<<<< run all test cases cdc_server_tips success! >>>>>> ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 7838 0 --:--:-- --:--:-- --:--:-- 8100 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:41:42.823","puller_resolved_ts":"2025-04-04 15:41:36.973","last_synced_ts":"2025-04-04 15:41:37.023","now_ts":"2025-04-04 15:41:44.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:42.823","puller_resolved_ts":"2025-04-04' '15:41:36.973","last_synced_ts":"2025-04-04' '15:41:37.023","now_ts":"2025-04-04' '15:41:44.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:42.823","puller_resolved_ts":"2025-04-04' '15:41:36.973","last_synced_ts":"2025-04-04' '15:41:37.023","now_ts":"2025-04-04' '15:41:44.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table test.t_36 exists table test.t_37 exists table test.t_38 exists table test.t_39 exists table test.t_40 exists table test.t_41 exists table test.t_42 exists table test.t_43 exists table test.t_44 exists table test.t_45 exists table test.t_46 exists table test.t_47 exists table test.t_48 exists table test.t_49 exists table test.t_50 exists table test.t_51 exists table test.t_52 exists table test.t_53 exists table test.t_54 exists table test.t_55 exists table test.t_56 exists table test.t_57 exists table test.t_58 exists table test.t_59 exists table test.t_60 exists table test.t_61 exists table test.t_62 exists table test.t_63 exists table test.t_64 exists table test.t_65 exists table test.t_66 exists table test.t_67 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 table ddl_manager.finish_mark not exists for 37-th check, retry later table test.t_68 exists table test.t_69 exists table test.t_70 exists table test.t_71 exists table test.t_72 exists table test.t_73 exists table test.t_74 exists table test.t_75 exists table test.t_76 exists table test.t_77 exists table test.t_78 exists table test.t_79 exists table test.t_80 exists table test.t_81 exists table test.t_82 exists table test.t_83 exists table test.t_84 exists table test.t_85 exists table test.t_86 exists table test.t_87 exists table test.t_88 exists table test.t_89 exists table test.t_90 exists table test.t_91 exists table test.t_92 exists table test.t_93 exists table test.t_94 exists table test.t_95 exists table test.t_96 exists table test.t_97 exists table test.t_98 exists table test.t_99 exists check diff failed 2-th time, retry later table test.t_100 exists You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide check diff failed 5-th time, retry later table test.t_100 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.12968.out cli tso query --pd=http://127.0.0.1:2379 table ddl_attributes.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 38-th check, retry later * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DAA9913E835 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:41:47 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created Exiting on signal: INTERRUPT wait process cdc.test exit for 2-th time... + set +x + tso='457114256948068353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114256948068353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:47 CST 2025] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1300213004.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check diff successfully [Fri Apr 4 15:41:47 CST 2025] <<<<<< run test case ddl_attributes success! >>>>>> wait process cdc.test exit for 1-th time... check diff successfully table test.t_100 not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:48 CST 2025] <<<<<< run test case region_merge success! >>>>>> table ddl_manager.finish_mark not exists for 39-th check, retry later table test.t_100 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:50 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41e0496a-90cf-4ba9-9ee9-318c67ee1529 {"id":"41e0496a-90cf-4ba9-9ee9-318c67ee1529","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8418d7 41e0496a-90cf-4ba9-9ee9-318c67ee1529 /tidb/cdc/default/default/upstream/7489359953913382495 {"id":7489359953913382495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41e0496a-90cf-4ba9-9ee9-318c67ee1529 {"id":"41e0496a-90cf-4ba9-9ee9-318c67ee1529","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8418d7 41e0496a-90cf-4ba9-9ee9-318c67ee1529 /tidb/cdc/default/default/upstream/7489359953913382495 {"id":7489359953913382495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41e0496a-90cf-4ba9-9ee9-318c67ee1529 {"id":"41e0496a-90cf-4ba9-9ee9-318c67ee1529","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752507} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf8418d7 41e0496a-90cf-4ba9-9ee9-318c67ee1529 /tidb/cdc/default/default/upstream/7489359953913382495 {"id":7489359953913382495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:41:50 CST 2025] <<<<<< START pulsar cluster in partition_table case >>>>>>' [Fri Apr 4 15:41:50 CST 2025] <<<<<< START pulsar cluster in partition_table case >>>>>> + workdir=/tmp/tidb_cdc_test/partition_table + cluster_type=normal + cd /tmp/tidb_cdc_test/partition_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/partition_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/partition_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/partition_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/partition_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/partition_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/partition_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/partition_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/partition_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/partition_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:41:50 CST 2025] <<<<<< START pulsar cluster in normal mode in partition_table case >>>>>>' [Fri Apr 4 15:41:50 CST 2025] <<<<<< START pulsar cluster in normal mode in partition_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/partition_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/partition_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/partition_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 40-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table test.t_100 not exists for 4-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } check diff failed 1-th time, retry later [Pipeline] // stage [Pipeline] } + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 41-th check, retry later table test.t_100 not exists for 5-th check, retry later check diff failed 2-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table ddl_manager.finish_mark not exists for 42-th check, retry later table test.t_100 not exists for 6-th check, retry later check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release public/default ++ date + echo '[Fri Apr 4 15:41:56 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:41:56 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.13469.out cli changefeed create --start-ts=457114256948068353 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-partition-table-test-31230?protocol=canal-json&enable-tidb-extension=true' table ddl_manager.finish_mark not exists for 43-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } Create changefeed successfully! ID: 6dd5f970-85f2-452d-a43c-75f1d4bb34d6 Info: {"upstream_id":7489359953913382495,"namespace":"default","id":"6dd5f970-85f2-452d-a43c-75f1d4bb34d6","sink_uri":"pulsar://127.0.0.1:6650/ticdc-partition-table-test-31230?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:41:57.971881216+08:00","start_ts":457114256948068353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114256948068353,"checkpoint_ts":457114256948068353,"checkpoint_time":"2025-04-04 15:41:46.058"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } check diff failed 4-th time, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv table test.t_100 not exists for 7-th check, retry later [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 44-th check, retry later + set +x [Fri Apr 4 15:41:59 CST 2025] <<<<<< START Pulsar consumer in partition_table case >>>>>> start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.t_100 not exists for 8-th check, retry later check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 45-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: pulsar... <<================= VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360026148248944 TiDB Cluster ID. tikv_gc_leader_uuid 657feff8af8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:9049, start at 2025-04-04 15:42:00.420114833 +0800 CST m=+1.648091664 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:00.430 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:00.432 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:00.432 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360026148248944 TiDB Cluster ID. tikv_gc_leader_uuid 657feff8af8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:9049, start at 2025-04-04 15:42:00.420114833 +0800 CST m=+1.648091664 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:00.430 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:00.432 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:00.432 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360026916769930 TiDB Cluster ID. tikv_gc_leader_uuid 657feff8afc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:9111, start at 2025-04-04 15:42:00.406258799 +0800 CST m=+1.568518324 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:00.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:00.383 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:00.383 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t_100 not exists for 9-th check, retry later check diff successfully table ddl_manager.finish_mark not exists for 46-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.10418.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table test.t_100 not exists for 10-th check, retry later + set +x + tso='457114261528772609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114261528772609 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:04 CST 2025] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.1046210464.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 47-th check, retry later check_changefeed_state http://127.0.0.1:2379 8d71fe77-92ff-4a8b-8d03-180a2a7a49fa finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=8d71fe77-92ff-4a8b-8d03-180a2a7a49fa + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 8d71fe77-92ff-4a8b-8d03-180a2a7a49fa -s + info='{ "upstream_id": 7489359554550992290, "namespace": "default", "id": "8d71fe77-92ff-4a8b-8d03-180a2a7a49fa", "state": "finished", "checkpoint_tso": 457114258999083011, "checkpoint_time": "2025-04-04 15:41:53.882", "error": null }' + echo '{ "upstream_id": 7489359554550992290, "namespace": "default", "id": "8d71fe77-92ff-4a8b-8d03-180a2a7a49fa", "state": "finished", "checkpoint_tso": 457114258999083011, "checkpoint_time": "2025-04-04 15:41:53.882", "error": null }' { "upstream_id": 7489359554550992290, "namespace": "default", "id": "8d71fe77-92ff-4a8b-8d03-180a2a7a49fa", "state": "finished", "checkpoint_tso": 457114258999083011, "checkpoint_time": "2025-04-04 15:41:53.882", "error": null } ++ echo '{' '"upstream_id":' 7489359554550992290, '"namespace":' '"default",' '"id":' '"8d71fe77-92ff-4a8b-8d03-180a2a7a49fa",' '"state":' '"finished",' '"checkpoint_tso":' 457114258999083011, '"checkpoint_time":' '"2025-04-04' '15:41:53.882",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7489359554550992290, '"namespace":' '"default",' '"id":' '"8d71fe77-92ff-4a8b-8d03-180a2a7a49fa",' '"state":' '"finished",' '"checkpoint_tso":' 457114258999083011, '"checkpoint_time":' '"2025-04-04' '15:41:53.882",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:00 CST 2025] <<<<<< run test case changefeed_finish success! >>>>>> table test.t_100 not exists for 11-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 48-th check, retry later table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360056707545356 TiDB Cluster ID. tikv_gc_leader_uuid 657feffee500007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:7522, start at 2025-04-04 15:42:06.752797565 +0800 CST m=+1.529450271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:06.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:06.740 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:06.740 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360056707545356 TiDB Cluster ID. tikv_gc_leader_uuid 657feffee500007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:7522, start at 2025-04-04 15:42:06.752797565 +0800 CST m=+1.529450271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:06.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:06.740 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:06.740 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360057871430362 TiDB Cluster ID. tikv_gc_leader_uuid 657feffefe00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:7585, start at 2025-04-04 15:42:06.847373988 +0800 CST m=+1.559793366 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:06.855 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:06.840 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:06.840 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.t2 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:08 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f6e4c1db-4f18-4e83-9131-b2077d26935d {"id":"f6e4c1db-4f18-4e83-9131-b2077d26935d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752525} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfc471d4 f6e4c1db-4f18-4e83-9131-b2077d26935d /tidb/cdc/default/default/upstream/7489360026148248944 {"id":7489360026148248944,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f6e4c1db-4f18-4e83-9131-b2077d26935d {"id":"f6e4c1db-4f18-4e83-9131-b2077d26935d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752525} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfc471d4 f6e4c1db-4f18-4e83-9131-b2077d26935d /tidb/cdc/default/default/upstream/7489360026148248944 {"id":7489360026148248944,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f6e4c1db-4f18-4e83-9131-b2077d26935d {"id":"f6e4c1db-4f18-4e83-9131-b2077d26935d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752525} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfc471d4 f6e4c1db-4f18-4e83-9131-b2077d26935d /tidb/cdc/default/default/upstream/7489360026148248944 {"id":7489360026148248944,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:42:08 CST 2025] <<<<<< START pulsar cluster in ddl_sequence case >>>>>>' [Fri Apr 4 15:42:08 CST 2025] <<<<<< START pulsar cluster in ddl_sequence case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_sequence + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_sequence + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_sequence/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_sequence/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_sequence/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_sequence/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_sequence/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_sequence/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_sequence/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_sequence/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_sequence/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:42:08 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_sequence case >>>>>>' [Fri Apr 4 15:42:08 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_sequence case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_sequence/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_sequence/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_sequence/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table test.t_100 not exists for 12-th check, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 49-th check, retry later [Fri Apr 4 15:42:09 CST 2025] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.88968898.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.t2 not exists for 3-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table test.t_100 not exists for 13-th check, retry later check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 50-th check, retry later table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 exists table processor_delay.t5 exists table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later table partition_table.t2 not exists for 4-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.t_100 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table batch_add_table.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:12 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51d4c3f6-e062-44b1-946a-633ba27f3d1e {"id":"51d4c3f6-e062-44b1-946a-633ba27f3d1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752529} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe193ce 51d4c3f6-e062-44b1-946a-633ba27f3d1e /tidb/cdc/default/default/upstream/7489360056707545356 {"id":7489360056707545356,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51d4c3f6-e062-44b1-946a-633ba27f3d1e {"id":"51d4c3f6-e062-44b1-946a-633ba27f3d1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752529} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe193ce 51d4c3f6-e062-44b1-946a-633ba27f3d1e /tidb/cdc/default/default/upstream/7489360056707545356 {"id":7489360056707545356,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51d4c3f6-e062-44b1-946a-633ba27f3d1e {"id":"51d4c3f6-e062-44b1-946a-633ba27f3d1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752529} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfe193ce 51d4c3f6-e062-44b1-946a-633ba27f3d1e /tidb/cdc/default/default/upstream/7489360056707545356 {"id":7489360056707545356,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:42:12 CST 2025] <<<<<< START pulsar cluster in event_filter case >>>>>>' [Fri Apr 4 15:42:12 CST 2025] <<<<<< START pulsar cluster in event_filter case >>>>>> + workdir=/tmp/tidb_cdc_test/event_filter + cluster_type=normal + cd /tmp/tidb_cdc_test/event_filter + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/event_filter/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/event_filter/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/event_filter/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/event_filter/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/event_filter/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/event_filter/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/event_filter/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/event_filter/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/event_filter/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:42:12 CST 2025] <<<<<< START pulsar cluster in normal mode in event_filter case >>>>>>' [Fri Apr 4 15:42:12 CST 2025] <<<<<< START pulsar cluster in normal mode in event_filter case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/event_filter/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/event_filter/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/event_filter/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 51-th check, retry later table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later table partition_table.t2 not exists for 5-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:42:14 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:42:14 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.10929.out cli changefeed create --start-ts=457114261528772609 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-9027?protocol=canal-json&enable-tidb-extension=true' + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff successfully table batch_add_table.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 52-th check, retry later wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: 3a495405-0c4d-466c-a497-1ce9aa289c81 Info: {"upstream_id":7489360026148248944,"namespace":"default","id":"3a495405-0c4d-466c-a497-1ce9aa289c81","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-9027?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:15.513607203+08:00","start_ts":457114261528772609,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114261528772609,"checkpoint_ts":457114261528772609,"checkpoint_time":"2025-04-04 15:42:03.532"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:16 CST 2025] <<<<<< run test case changefeed_pause_resume success! >>>>>> table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table batch_add_table.finish_mark not exists for 3-th check, retry later + set +x [Fri Apr 4 15:42:16 CST 2025] <<<<<< START Pulsar consumer in ddl_sequence case >>>>>> table ddl_manager.finish_mark not exists for 53-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later public/default table batch_add_table.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 54-th check, retry later ++ date + echo '[Fri Apr 4 15:42:19 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:42:19 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.9365.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-event-filter-1547?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later Create changefeed successfully! ID: 2908aa5a-7d60-41eb-bc84-f92b84323812 Info: {"upstream_id":7489360056707545356,"namespace":"default","id":"2908aa5a-7d60-41eb-bc84-f92b84323812","sink_uri":"pulsar://127.0.0.1:6650/ticdc-event-filter-1547?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:19.925019502+08:00","start_ts":457114265673007107,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_name*"],"ignore_event":["rename table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114265673007107,"checkpoint_ts":457114265673007107,"checkpoint_time":"2025-04-04 15:42:19.341"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table partition_table.finish_mark not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table batch_add_table.finish_mark not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 55-th check, retry later + set +x [Fri Apr 4 15:42:21 CST 2025] <<<<<< START Pulsar consumer in event_filter case >>>>>> table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 6-th check, retry later table ddl_manager.finish_mark not exists for 56-th check, retry later table ddl_sequence.finish_mark not exists for 1-th check, retry later table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360128298166427 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00ff980009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:4972, start at 2025-04-04 15:42:24.245046279 +0800 CST m=+1.780448579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.253 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.230 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.230 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table partition_table.finish_mark not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360128298166427 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00ff980009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:4972, start at 2025-04-04 15:42:24.245046279 +0800 CST m=+1.780448579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.253 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.230 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.230 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360114954652232 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00fec4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k6tq-4093j, pid:5047, start at 2025-04-04 15:42:24.224944275 +0800 CST m=+1.690623892 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:24.232 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:24.227 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:24.227 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table batch_add_table.finish_mark not exists for 7-th check, retry later table ddl_manager.finish_mark not exists for 57-th check, retry later table ddl_sequence.finish_mark not exists for 2-th check, retry later table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table partition_table.finish_mark not exists for 6-th check, retry later table event_filter.t1 does not exists table event_filter.t1 exists table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 8-th check, retry later table ddl_manager.finish_mark not exists for 58-th check, retry later table ddl_sequence.finish_mark not exists for 3-th check, retry later [Fri Apr 4 15:42:26 CST 2025] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.64076409.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later table partition_table.finish_mark exists check diff successfully table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 59-th check, retry later table ddl_sequence.finish_mark not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:30 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cdfcc83d-c70a-4c46-ba81-05d82ef7b90e {"id":"cdfcc83d-c70a-4c46-ba81-05d82ef7b90e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752547} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02102cf cdfcc83d-c70a-4c46-ba81-05d82ef7b90e /tidb/cdc/default/default/upstream/7489360128298166427 {"id":7489360128298166427,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cdfcc83d-c70a-4c46-ba81-05d82ef7b90e {"id":"cdfcc83d-c70a-4c46-ba81-05d82ef7b90e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752547} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02102cf cdfcc83d-c70a-4c46-ba81-05d82ef7b90e /tidb/cdc/default/default/upstream/7489360128298166427 {"id":7489360128298166427,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cdfcc83d-c70a-4c46-ba81-05d82ef7b90e {"id":"cdfcc83d-c70a-4c46-ba81-05d82ef7b90e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752547} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc02102cf cdfcc83d-c70a-4c46-ba81-05d82ef7b90e /tidb/cdc/default/default/upstream/7489360128298166427 {"id":7489360128298166427,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:42:30 CST 2025] <<<<<< START pulsar cluster in force_replicate_table case >>>>>>' [Fri Apr 4 15:42:30 CST 2025] <<<<<< START pulsar cluster in force_replicate_table case >>>>>> + workdir=/tmp/tidb_cdc_test/force_replicate_table + cluster_type=normal + cd /tmp/tidb_cdc_test/force_replicate_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/force_replicate_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/force_replicate_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/force_replicate_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/force_replicate_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:29 CST 2025] <<<<<< run test case partition_table success! >>>>>> + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/force_replicate_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/force_replicate_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/force_replicate_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/force_replicate_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/force_replicate_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:42:30 CST 2025] <<<<<< START pulsar cluster in normal mode in force_replicate_table case >>>>>>' [Fri Apr 4 15:42:30 CST 2025] <<<<<< START pulsar cluster in normal mode in force_replicate_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/force_replicate_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/force_replicate_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/force_replicate_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table event_filter.t_alter exists table event_filter.t_name not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 10-th check, retry later table ddl_manager.finish_mark not exists for 60-th check, retry later table ddl_sequence.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark not exists for 11-th check, retry later [2025/04/04 15:42:24.313 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:42:24.313 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489359566708424978] [2025/04/04 15:42:24.314 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:42:24.314 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:42:24.314 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:42:24.315 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:42:24.315 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489359566708424978] [2025/04/04 15:42:24.315 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:42:24.315 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:42:24.316 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:42:24.316 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:42:24.316 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:42:24.316 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:42:24.316 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:42:24.316 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:42:24.316 +08:00] [INFO] [tikv_driver.go:201] ["using API V1."] [2025/04/04 15:42:24.317 +08:00] [INFO] [main.go:180] ["genLock started"] [2025/04/04 15:42:24.318 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=5] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:42:26.317 +08:00] [INFO] [pd.go:441] ["adaptive update ts interval state transition"] [configuredInterval=2s] [prevAdaptiveUpdateInterval=2s] [newAdaptiveUpdateInterval=2s] [requiredStaleness=0s] [prevState=unknown(0)] [newState=normal] table event_filter.t_name exists table event_filter.t_name1 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:32 CST 2025] <<<<<< run test case ddl_sequence success! >>>>>> table ddl_manager.finish_mark not exists for 61-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later [2025/04/04 15:42:34.318 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=1] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:42:34.318 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=4] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:42:34.318 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=134] [addr=127.0.0.1:9500] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:42:34.327 +08:00] [INFO] [main.go:196] ["genLock done"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_dispatcher.go:490] ["[tso] exit tso connection contexts updater"] [dc-location=global] [2025/04/04 15:42:34.327 +08:00] [INFO] [resource_manager_client.go:296] ["[resource manager] exit resource token dispatcher"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_dispatcher.go:264] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2025/04/04 15:42:34.327 +08:00] [INFO] [pd_service_discovery.go:551] ["[pd] exit member loop due to context canceled"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_client.go:147] ["[tso] exit tso dispatcher check loop"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_dispatcher.go:201] ["[tso] exit tso dispatcher"] [dc-location=global] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_dispatcher.go:158] ["[tso] exit tso deadline watcher"] [dc-location=global] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_client.go:157] ["[tso] closing tso client"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_client.go:162] ["[tso] close tso client"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_stream.go:359] ["tsoStream.recvLoop ended"] [stream=127.0.0.1:2379-1] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241111073742-238d4d79ea31/tso_stream.go:427\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700"] [2025/04/04 15:42:34.327 +08:00] [INFO] [tso_client.go:164] ["[tso] tso client is closed"] [2025/04/04 15:42:34.328 +08:00] [INFO] [pd_service_discovery.go:644] ["[pd] close pd service discovery client"] + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table event_filter.t_name1 exists table event_filter.t_name2 not exists for 1-th check, retry later Starting Upstream TiDB... table batch_add_table.finish_mark not exists for 12-th check, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 62-th check, retry later table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:42:36 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:42:36 CST 2025] <<<<<< pulsar is ready >>>>>> table event_filter.t_name2 exists table event_filter.t_name3 not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 13-th check, retry later Create changefeed successfully! ID: d5a4344d-547b-4a4f-a548-caee81fd8e5b Info: {"upstream_id":7489360128298166427,"namespace":"default","id":"d5a4344d-547b-4a4f-a548-caee81fd8e5b","sink_uri":"pulsar://127.0.0.1:6650/ticdc-force_replicate_table-test-5570?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:37.100704847+08:00","start_ts":457114267662417921,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114267662417921,"checkpoint_ts":457114267662417921,"checkpoint_time":"2025-04-04 15:42:26.930"} [Fri Apr 4 15:42:37 CST 2025] <<<<<< START Pulsar consumer in force_replicate_table case >>>>>> table ddl_manager.finish_mark not exists for 63-th check, retry later table force_replicate_table.t0 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360181691304965 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01c0c4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:11260, start at 2025-04-04 15:42:36.610725752 +0800 CST m=+1.554478496 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:36.618 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:36.593 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:36.593 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360181691304965 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01c0c4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:11260, start at 2025-04-04 15:42:36.610725752 +0800 CST m=+1.554478496 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:36.618 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:36.593 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:36.593 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360183122653973 TiDB Cluster ID. tikv_gc_leader_uuid 657ff01c2380007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:11345, start at 2025-04-04 15:42:36.695543368 +0800 CST m=+1.569445576 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:36.703 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:36.686 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:36.686 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later table event_filter.t_name3 exists table batch_add_table.finish_mark not exists for 14-th check, retry later table ddl_manager.finish_mark not exists for 64-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12738.out cli tso query --pd=http://127.0.0.1:2379 table test.t2 not exists for 1-th check, retry later table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 15-th check, retry later table ddl_manager.finish_mark not exists for 65-th check, retry later + set +x + tso='457114271034376193 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114271034376193 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:41 CST 2025] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1279312795.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table event_filter.finish_mark exists check diff failed 1-th time, retry later table force_replicate_table.t1 exists table force_replicate_table.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_with_exists/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:42:40 CST 2025] <<<<<< run test case ddl_with_exists success! >>>>>> table test.t2 not exists for 2-th check, retry later table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table batch_add_table.finish_mark not exists for 16-th check, retry later table ddl_manager.finish_mark not exists for 66-th check, retry later check diff failed 2-th time, retry later table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later table test.t2 not exists for 3-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:44 GMT < Content-Length: 861 < Content-Type: text/plain; charset=utf-8 < { [861 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/806537c9-eced-4051-bca3-cd581f659828 {"id":"806537c9-eced-4051-bca3-cd581f659828","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752561} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc057efdb 806537c9-eced-4051-bca3-cd581f659828 /tidb/cdc/default/default/upstream/7489360181691304965 {"id":7489360181691304965,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/806537c9-eced-4051-bca3-cd581f659828 {"id":"806537c9-eced-4051-bca3-cd581f659828","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752561} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc057efdb 806537c9-eced-4051-bca3-cd581f659828 /tidb/cdc/default/default/upstream/7489360181691304965 {"id":7489360181691304965,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/806537c9-eced-4051-bca3-cd581f659828 {"id":"806537c9-eced-4051-bca3-cd581f659828","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752561} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc057efdb 806537c9-eced-4051-bca3-cd581f659828 /tidb/cdc/default/default/upstream/7489360181691304965 {"id":7489360181691304965,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:42:44 CST 2025] <<<<<< START pulsar cluster in cli_with_auth case >>>>>>' [Fri Apr 4 15:42:44 CST 2025] <<<<<< START pulsar cluster in cli_with_auth case >>>>>> + workdir=/tmp/tidb_cdc_test/cli_with_auth + cluster_type=normal + cd /tmp/tidb_cdc_test/cli_with_auth + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/cli_with_auth/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/cli_with_auth/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/cli_with_auth/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/cli_with_auth/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/cli_with_auth/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/cli_with_auth/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/cli_with_auth/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/cli_with_auth/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/cli_with_auth/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:42:44 CST 2025] <<<<<< START pulsar cluster in normal mode in cli_with_auth case >>>>>>' [Fri Apr 4 15:42:44 CST 2025] <<<<<< START pulsar cluster in normal mode in cli_with_auth case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/cli_with_auth/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/cli_with_auth/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/cli_with_auth/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table batch_add_table.finish_mark not exists for 17-th check, retry later table ddl_manager.finish_mark not exists for 67-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table test.t2 not exists for 4-th check, retry later [Pipeline] // node [Pipeline] } table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later [Pipeline] // podTemplate [Pipeline] } check diff successfully wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark exists table test.t_1 does not exists check diff successfully table ddl_manager.finish_mark not exists for 68-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:47 CST 2025] <<<<<< run test case event_filter success! >>>>>> wait process cdc.test exit for 1-th time... table test.t2 not exists for 5-th check, retry later table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later Verifying downstream PD is started... wait process cdc.test exit for 2-th time... table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:48 CST 2025] <<<<<< run test case batch_add_table success! >>>>>> + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 69-th check, retry later table test.t2 not exists for 6-th check, retry later table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:42:50 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:42:50 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13254.out cli changefeed create --start-ts=457114271034376193 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 70-th check, retry later [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360181691304965,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:51.747301047+08:00","start_ts":457114271034376193,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114271034376193,"checkpoint_ts":457114271034376193,"checkpoint_time":"2025-04-04 15:42:39.793"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 table test.t2 not exists for 7-th check, retry later table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 + set +x [Fri Apr 4 15:42:53 CST 2025] <<<<<< START Pulsar consumer in cli_with_auth case >>>>>> table test.simple not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360244544906433 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02b97c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:12112, start at 2025-04-04 15:42:52.557771029 +0800 CST m=+1.632974552 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.561 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.561 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360244544906433 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02b97c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:12112, start at 2025-04-04 15:42:52.557771029 +0800 CST m=+1.632974552 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.561 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.561 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360249650962168 TiDB Cluster ID. tikv_gc_leader_uuid 657ff02bb10000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-k7ht-l4r36, pid:12198, start at 2025-04-04 15:42:52.649224417 +0800 CST m=+1.649049714 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:52.657 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:52.662 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:52.662 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 table ddl_manager.finish_mark not exists for 71-th check, retry later run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 table test.t2 exists check diff successfully table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.13633.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 72-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:55 CST 2025] <<<<<< run test case resolve_lock success! >>>>>> run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t36 exists table processor_delay.t37 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... + set +x + tso='457114275207184385 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114275207184385 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:57 CST 2025] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1366713669.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark exists check diff successfully table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114271427330072, "checkpoint_time": "2025-04-04 15:42:41.292", "error": null }' + echo '{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114271427330072, "checkpoint_time": "2025-04-04 15:42:41.292", "error": null }' { "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114271427330072, "checkpoint_time": "2025-04-04 15:42:41.292", "error": null } ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114271427330072, '"checkpoint_time":' '"2025-04-04' '15:42:41.292",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114271427330072, '"checkpoint_time":' '"2025-04-04' '15:42:41.292",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 wait process cdc.test exit for 1-th time... changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: pulsar... <<================= Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0034e1cb8} {Type:update Path:[Config SyncPointRetention] From: To:0xc0034e1cd0} {Type:update Path:[Config Consistent] From: To:0xc000a74770} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13600.out cli changefeed --changefeed-id custom-changefeed-name pause wait process cdc.test exit for 3-th time... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 4-th time... run task successfully wait process cdc.test exit for 1-th time... table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later wait process cdc.test exit for 5-th time... wait process cdc.test exit for 6-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:00 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39eb0589-b10e-48b6-8170-2c57274fcaab {"id":"39eb0589-b10e-48b6-8170-2c57274fcaab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc095eed9 39eb0589-b10e-48b6-8170-2c57274fcaab /tidb/cdc/default/default/upstream/7489360244544906433 {"id":7489360244544906433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39eb0589-b10e-48b6-8170-2c57274fcaab {"id":"39eb0589-b10e-48b6-8170-2c57274fcaab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc095eed9 39eb0589-b10e-48b6-8170-2c57274fcaab /tidb/cdc/default/default/upstream/7489360244544906433 {"id":7489360244544906433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39eb0589-b10e-48b6-8170-2c57274fcaab {"id":"39eb0589-b10e-48b6-8170-2c57274fcaab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752577} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc095eed9 39eb0589-b10e-48b6-8170-2c57274fcaab /tidb/cdc/default/default/upstream/7489360244544906433 {"id":7489360244544906433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:43:00 CST 2025] <<<<<< START pulsar cluster in resourcecontrol case >>>>>>' [Fri Apr 4 15:43:00 CST 2025] <<<<<< START pulsar cluster in resourcecontrol case >>>>>> + workdir=/tmp/tidb_cdc_test/resourcecontrol + cluster_type=normal + cd /tmp/tidb_cdc_test/resourcecontrol + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/resourcecontrol/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/resourcecontrol/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/resourcecontrol/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/resourcecontrol/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/resourcecontrol/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/resourcecontrol/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/resourcecontrol/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/resourcecontrol/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/resourcecontrol/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:43:00 CST 2025] <<<<<< START pulsar cluster in normal mode in resourcecontrol case >>>>>>' [Fri Apr 4 15:43:00 CST 2025] <<<<<< START pulsar cluster in normal mode in resourcecontrol case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/resourcecontrol/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/resourcecontrol/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/resourcecontrol/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + set +x wait process cdc.test exit for 7-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:00 CST 2025] <<<<<< run test case force_replicate_table success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 8-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ci_collation_compatibility/run.sh using Sink-Type: pulsar... <<================= [Fri Apr 4 15:43:01 CST 2025] <<<<<< run test case ci_collation_compatibility success! >>>>>> cdc.test: no process found wait process cdc.test exit for 9-th time... process cdc.test already exit [Fri Apr 4 15:43:01 CST 2025] <<<<<< run test case ddl_manager success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114275831611394, "checkpoint_time": "2025-04-04 15:42:58.093", "error": null }' + echo '{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114275831611394, "checkpoint_time": "2025-04-04 15:42:58.093", "error": null }' { "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114275831611394, "checkpoint_time": "2025-04-04 15:42:58.093", "error": null } ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114275831611394, '"checkpoint_time":' '"2025-04-04' '15:42:58.093",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114275831611394, '"checkpoint_time":' '"2025-04-04' '15:42:58.093",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13677.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc003d41c88} {Type:update Path:[Config SyncPointRetention] From: To:0xc003d41c98} {Type:update Path:[Config Consistent] From: To:0xc000355b90} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360181691304965,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:42:51.747301047+08:00","start_ts":457114271034376193,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","encoder_concurrency":32,"terminator":"\r\n","enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":0,"checkpoint_ts":457114275831611394,"checkpoint_time":"2025-04-04 15:42:58.093"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13716.out cli changefeed --changefeed-id custom-changefeed-name resume [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } public/default ++ date + echo '[Fri Apr 4 15:43:06 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:43:06 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.14132.out cli changefeed create --start-ts=457114275207184385 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-1728?protocol=canal-json&enable-tidb-extension=true' [Pipeline] // stage [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308457274525 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039a280002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10869, start at 2025-04-04 15:43:06.891007688 +0800 CST m=+1.451218716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Create changefeed successfully! ID: 01c32114-8aed-4ab8-b66d-c1984e738945 Info: {"upstream_id":7489360244544906433,"namespace":"default","id":"01c32114-8aed-4ab8-b66d-c1984e738945","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-1728?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:07.69561854+08:00","start_ts":457114275207184385,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114275207184385,"checkpoint_ts":457114275207184385,"checkpoint_time":"2025-04-04 15:42:55.711"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later + set +x [Fri Apr 4 15:43:09 CST 2025] <<<<<< START Pulsar consumer in resourcecontrol case >>>>>> table resourcecontrol.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.12277.out cli tso query --pd=http://127.0.0.1:2379 + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114278977077252, "checkpoint_time": "2025-04-04 15:43:10.092", "error": null }' + echo '{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114278977077252, "checkpoint_time": "2025-04-04 15:43:10.092", "error": null }' { "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114278977077252, "checkpoint_time": "2025-04-04 15:43:10.092", "error": null } ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114278977077252, '"checkpoint_time":' '"2025-04-04' '15:43:10.092",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114278977077252, '"checkpoint_time":' '"2025-04-04' '15:43:10.092",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13807.out cli changefeed --changefeed-id custom-changefeed-name remove table resourcecontrol.finish_mark exists + set +x + tso='457114278989922305 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114278989922305 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114278989922305 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:43:11 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1231412316.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: pulsar... <<================= wait process cdc.test exit for 1-th time... Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114279239483398 SinkURI: pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Fri Apr 4 15:43:13 CST 2025] <<<<<< run test case resourcecontrol success! >>>>>> [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } + set +x [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:14 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114278989922305 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.12375.out cli changefeed create --start-ts=457114278989922305 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360313431937432,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:15.081245786+08:00","start_ts":457114278989922305,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114278989922305,"checkpoint_ts":457114278989922305,"checkpoint_time":"2025-04-04 15:43:10.141"} PASS Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/vector/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:12 CST 2025] <<<<<< run test case ddl_puller_lag success! >>>>>> coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 + '[' pulsar == mysql ']' + stop_tidb_cluster changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13870.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360181691304965,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-15706?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:16.960080571+08:00","start_ts":457114280746811398,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114280746811398,"checkpoint_ts":457114280746811398,"checkpoint_time":"2025-04-04 15:43:16.843"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360357906585591 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0437e00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:24996, start at 2025-04-04 15:43:17.016666317 +0800 CST m=+1.793658350 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.025 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:16.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:16.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360357906585591 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0437e00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:24996, start at 2025-04-04 15:43:17.016666317 +0800 CST m=+1.793658350 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.025 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:16.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:16.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360344135264661 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0439780002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:25055, start at 2025-04-04 15:43:17.087407292 +0800 CST m=+1.797611824 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:17.097 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:17.086 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:17.086 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later + set +x start tidb cluster in /tmp/tidb_cdc_test/vector Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.26432.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later + set +x + tso='457114281570205697 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114281570205697 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114281860923399, "checkpoint_time": "2025-04-04 15:43:21.093", "error": null }' + echo '{ "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114281860923399, "checkpoint_time": "2025-04-04 15:43:21.093", "error": null }' { "upstream_id": 7489360181691304965, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114281860923399, "checkpoint_time": "2025-04-04 15:43:21.093", "error": null } ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114281860923399, '"checkpoint_time":' '"2025-04-04' '15:43:21.093",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360181691304965, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114281860923399, '"checkpoint_time":' '"2025-04-04' '15:43:21.093",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13981.out cli unsafe delete-service-gc-safepoint ***************** properties ***************** "readproportion"="0" "recordcount"="10000" "mysql.host"="127.0.0.1" "mysql.user"="root" "threadcount"="10" "requestdistribution"="uniform" "updateproportion"="0" "scanproportion"="0" "mysql.db"="move_table" "insertproportion"="0" "workload"="core" "mysql.port"="4000" "readallfields"="true" "dotransactions"="false" "operationcount"="0" ********************************************** Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14008.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } Run finished, takes 1.474911303s INSERT - Takes(s): 1.5, Count: 10000, OPS: 6807.0, Avg(us): 1432, Min(us): 944, Max(us): 7528, 95th(us): 2000, 99th(us): 3000 [Fri Apr 4 15:43:23 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2651526517.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // node reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table processor_delay.t50 exists [Pipeline] // stage [Pipeline] } check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360383910153533 TiDB Cluster ID. tikv_gc_leader_uuid 657ff04c6c8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:8305, start at 2025-04-04 15:43:26.148493768 +0800 CST m=+1.519158514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:26.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:26.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:26.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360383910153533 TiDB Cluster ID. tikv_gc_leader_uuid 657ff04c6c8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:8305, start at 2025-04-04 15:43:26.148493768 +0800 CST m=+1.519158514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:26.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:26.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:26.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360395799786317 TiDB Cluster ID. tikv_gc_leader_uuid 657ff04c8ec0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-frs0-qnz2g, pid:8385, start at 2025-04-04 15:43:26.275465126 +0800 CST m=+1.581985896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:26.283 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:26.267 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:26.267 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/vector/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/vector/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/vector/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/vector/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/vector/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:27 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:43:27 CST 2025] <<<<<< START pulsar cluster in move_table case >>>>>>' [Fri Apr 4 15:43:27 CST 2025] <<<<<< START pulsar cluster in move_table case >>>>>> + workdir=/tmp/tidb_cdc_test/move_table + cluster_type=normal + cd /tmp/tidb_cdc_test/move_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/move_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/move_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/move_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/move_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/move_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/move_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/move_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/move_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/move_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:43:27 CST 2025] <<<<<< START pulsar cluster in normal mode in move_table case >>>>>>' [Fri Apr 4 15:43:27 CST 2025] <<<<<< START pulsar cluster in normal mode in move_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/move_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/move_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/move_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14115.out cli unsafe resolve-lock --region=104 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:28 CST 2025] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.9675.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14146.out cli unsafe resolve-lock --region=104 --ts=457114283079892996 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114284020203521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114284020203521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:30 CST 2025] <<<<<< START cdc server in vector case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.97099711.out server --log-file /tmp/tidb_cdc_test/vector/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/vector/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 207k 6000 --:--:-- --:--:-- --:--:-- 212k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360422517814101 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0523d0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:8092, start at 2025-04-04 15:43:32.110085041 +0800 CST m=+1.695307180 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.118 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.084 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.084 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360422517814101 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0523d0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:8092, start at 2025-04-04 15:43:32.110085041 +0800 CST m=+1.695307180 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.118 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.084 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.084 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360415090763869 TiDB Cluster ID. tikv_gc_leader_uuid 657ff052554000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:8173, start at 2025-04-04 15:43:32.204381579 +0800 CST m=+1.725373979 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:32.212 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:32.181 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:32.181 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } public/default ++ date + echo '[Fri Apr 4 15:43:33 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:43:33 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.26982.out cli changefeed create --start-ts=457114281570205697 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:33 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ec9116-9604-40aa-95ab-6ae43714b5cf {"id":"81ec9116-9604-40aa-95ab-6ae43714b5cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752611} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc113cbde 81ec9116-9604-40aa-95ab-6ae43714b5cf /tidb/cdc/default/default/upstream/7489360383910153533 {"id":7489360383910153533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ec9116-9604-40aa-95ab-6ae43714b5cf {"id":"81ec9116-9604-40aa-95ab-6ae43714b5cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752611} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc113cbde 81ec9116-9604-40aa-95ab-6ae43714b5cf /tidb/cdc/default/default/upstream/7489360383910153533 {"id":7489360383910153533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ec9116-9604-40aa-95ab-6ae43714b5cf {"id":"81ec9116-9604-40aa-95ab-6ae43714b5cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752611} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc113cbde 81ec9116-9604-40aa-95ab-6ae43714b5cf /tidb/cdc/default/default/upstream/7489360383910153533 {"id":7489360383910153533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.9766.out cli changefeed create --start-ts=457114284020203521 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: b4851339-3cfb-4f87-b4fb-bd6f153419fd Info: {"upstream_id":7489360357906585591,"namespace":"default","id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink_uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:34.430801636+08:00","start_ts":457114281570205697,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114281570205697,"checkpoint_ts":457114281570205697,"checkpoint_time":"2025-04-04 15:43:19.984"} PASS Create changefeed successfully! ID: f380f47e-f4c8-4b69-9d27-9edf95b2a7fd Info: {"upstream_id":7489360383910153533,"namespace":"default","id":"f380f47e-f4c8-4b69-9d27-9edf95b2a7fd","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:34.336632021+08:00","start_ts":457114284020203521,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114284020203521,"checkpoint_ts":457114284020203521,"checkpoint_time":"2025-04-04 15:43:29.330"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:43:34 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.96389640.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 178k 0 --:--:-- --:--:-- --:--:-- 178k { "version": "v9.0.0-beta.1-10-ge8bfbe4ba", "git_hash": "e8bfbe4ba98e05140ab733ee3d08dd3b99918721", "id": "7eb7691b-12f4-4a06-9593-71efeb626f5a", "pid": 12798, "is_owner": true + set +x + set +x [Fri Apr 4 15:43:35 CST 2025] <<<<<< START Pulsar consumer in move_table case >>>>>> [Fri Apr 4 15:43:35 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2703927045.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:36 CST 2025] <<<<<< run test case cli_with_auth success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:37 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e58a5351-4a58-4f32-8dab-456d7e6849fa {"id":"e58a5351-4a58-4f32-8dab-456d7e6849fa","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13245cf e58a5351-4a58-4f32-8dab-456d7e6849fa /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e58a5351-4a58-4f32-8dab-456d7e6849fa {"id":"e58a5351-4a58-4f32-8dab-456d7e6849fa","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13245cf e58a5351-4a58-4f32-8dab-456d7e6849fa /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e58a5351-4a58-4f32-8dab-456d7e6849fa {"id":"e58a5351-4a58-4f32-8dab-456d7e6849fa","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13245cf e58a5351-4a58-4f32-8dab-456d7e6849fa /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:43:37 CST 2025] <<<<<< START pulsar cluster in ddl_only_block_related_table case >>>>>>' [Fri Apr 4 15:43:37 CST 2025] <<<<<< START pulsar cluster in ddl_only_block_related_table case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_only_block_related_table + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_only_block_related_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:43:37 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_only_block_related_table case >>>>>>' [Fri Apr 4 15:43:37 CST 2025] <<<<<< START pulsar cluster in normal mode in ddl_only_block_related_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:38 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4818 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001a6e1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282094755854 MinTableBarrierTs:457114286354595853 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282094755854,"min-table-barrier-ts":457114286354595853,"admin-job-type":0} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001a6e1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282094755854 MinTableBarrierTs:457114286354595853 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282094755854,"min-table-barrier-ts":457114286354595853,"admin-job-type":0} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001a6e1b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282094755854 MinTableBarrierTs:457114286354595853 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282094755854,"min-table-barrier-ts":457114286354595853,"admin-job-type":0} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Fri Apr 4 15:43:39 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2714527147.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308457274525 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039a280002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10869, start at 2025-04-04 15:43:06.891007688 +0800 CST m=+1.451218716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e16990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282277994559 MinTableBarrierTs:457114287166980097 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2891278e-8995-4bc9-a51c-1c8d2d8edcce {"id":"2891278e-8995-4bc9-a51c-1c8d2d8edcce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef17b3 2891278e-8995-4bc9-a51c-1c8d2d8edcce /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282277994667,"min-table-barrier-ts":457114287166980097,"admin-job-type":0} /tidb/cdc/default/default/task/position/2891278e-8995-4bc9-a51c-1c8d2d8edcce/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e16990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282277994559 MinTableBarrierTs:457114287166980097 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2891278e-8995-4bc9-a51c-1c8d2d8edcce {"id":"2891278e-8995-4bc9-a51c-1c8d2d8edcce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef17b3 2891278e-8995-4bc9-a51c-1c8d2d8edcce /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282277994667,"min-table-barrier-ts":457114287166980097,"admin-job-type":0} /tidb/cdc/default/default/task/position/2891278e-8995-4bc9-a51c-1c8d2d8edcce/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'failed to get info:' /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b4851339-3cfb-4f87-b4fb-bd6f153419fd {UpstreamID:7489360357906585591 Namespace:default ID:b4851339-3cfb-4f87-b4fb-bd6f153419fd SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:34.430801636 +0800 CST StartTs:457114281570205697 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e16990 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114285345079302} {CheckpointTs:457114282277994559 MinTableBarrierTs:457114287166980097 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2891278e-8995-4bc9-a51c-1c8d2d8edcce {"id":"2891278e-8995-4bc9-a51c-1c8d2d8edcce","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752619} /tidb/cdc/default/__cdc_meta__/capture/6dd1b716-6705-4cb8-b5b1-7a132e98bf99 {"id":"6dd1b716-6705-4cb8-b5b1-7a132e98bf99","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752616} /tidb/cdc/default/__cdc_meta__/capture/f2ee8e9d-25ce-4230-8aba-e199601192fe {"id":"f2ee8e9d-25ce-4230-8aba-e199601192fe","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752604} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef16fb f2ee8e9d-25ce-4230-8aba-e199601192fe /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef177e 6dd1b716-6705-4cb8-b5b1-7a132e98bf99 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ef17b3 2891278e-8995-4bc9-a51c-1c8d2d8edcce /tidb/cdc/default/default/changefeed/info/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"upstream-id":7489360357906585591,"namespace":"default","changefeed-id":"b4851339-3cfb-4f87-b4fb-bd6f153419fd","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-7046?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:34.430801636+08:00","start-ts":457114281570205697,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114285345079302} /tidb/cdc/default/default/changefeed/status/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":457114282277994667,"min-table-barrier-ts":457114287166980097,"admin-job-type":0} /tidb/cdc/default/default/task/position/2891278e-8995-4bc9-a51c-1c8d2d8edcce/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6dd1b716-6705-4cb8-b5b1-7a132e98bf99/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f2ee8e9d-25ce-4230-8aba-e199601192fe/b4851339-3cfb-4f87-b4fb-bd6f153419fd {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360357906585591 {"id":7489360357906585591,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table move_table.usertable exists go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/go-mysql-org/go-mysql v1.9.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 table test.finish_mark not exists for 1-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:43:44 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:43:44 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.10112.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table table test.finish_mark exists go: downloading github.com/ardielle/ardielle-go v1.5.2 check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7489360422517814101,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:43:45.162195376+08:00","start_ts":457114288018948098,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114288018948098,"checkpoint_ts":457114288018948098,"checkpoint_time":"2025-04-04 15:43:44.584"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:46 CST 2025] <<<<<< run test case vector success! >>>>>> + set +x [Fri Apr 4 15:43:46 CST 2025] <<<<<< START Pulsar consumer in ddl_only_block_related_table case >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360491614675491 TiDB Cluster ID. tikv_gc_leader_uuid 657ff062054000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:8775, start at 2025-04-04 15:43:48.279377387 +0800 CST m=+1.559406734 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:48.290 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:48.294 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:48.294 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360491614675491 TiDB Cluster ID. tikv_gc_leader_uuid 657ff062054000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:8775, start at 2025-04-04 15:43:48.279377387 +0800 CST m=+1.559406734 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:48.290 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:48.294 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:48.294 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360494338130187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0621a40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-xz56-4gxfc, pid:8839, start at 2025-04-04 15:43:48.343558394 +0800 CST m=+1.553781283 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:48.351 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:48.329 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:48.329 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... [Fri Apr 4 15:43:51 CST 2025] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.1029410296.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:54 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c330650-d73d-4fb0-b250-ebe4727e8255 {"id":"8c330650-d73d-4fb0-b250-ebe4727e8255","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17008cf 8c330650-d73d-4fb0-b250-ebe4727e8255 /tidb/cdc/default/default/upstream/7489360491614675491 {"id":7489360491614675491,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c330650-d73d-4fb0-b250-ebe4727e8255 {"id":"8c330650-d73d-4fb0-b250-ebe4727e8255","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17008cf 8c330650-d73d-4fb0-b250-ebe4727e8255 /tidb/cdc/default/default/upstream/7489360491614675491 {"id":7489360491614675491,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c330650-d73d-4fb0-b250-ebe4727e8255 {"id":"8c330650-d73d-4fb0-b250-ebe4727e8255","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752631} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17008cf 8c330650-d73d-4fb0-b250-ebe4727e8255 /tidb/cdc/default/default/upstream/7489360491614675491 {"id":7489360491614675491,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:43:54 CST 2025] <<<<<< START Pulsar consumer in sink_hang case >>>>>> check_changefeed_status 127.0.0.1:8300 a0dd4b26-139b-48aa-995a-6ed4f51acc15 normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=a0dd4b26-139b-48aa-995a-6ed4f51acc15 + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/a0dd4b26-139b-48aa-995a-6ed4f51acc15/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1131 0 --:--:-- --:--:-- --:--:-- 1131 + info='{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' + echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' {"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681} ++ echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 a0dd4b26-139b-48aa-995a-6ed4f51acc15 normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=a0dd4b26-139b-48aa-995a-6ed4f51acc15 + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/a0dd4b26-139b-48aa-995a-6ed4f51acc15/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1131 0 --:--:-- --:--:-- --:--:-- 1131 + info='{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' + echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' {"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681} ++ echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":457114290577735681,"checkpoint_ts":457114290577735681}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark exists wait process 9643 exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } Verifying downstream PD is started... [Pipeline] // timeout [Pipeline] } wait process 9643 exit for 2-th time... [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9643) - No such process wait process 9643 exit for 3-th time... process 9643 already exit [Fri Apr 4 15:43:56 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.1025110253.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later [2025/04/04 15:43:56.499 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:43:56.501 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:43:56.502 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=f2ee8e9d-25ce-4230-8aba-e199601192fe] [2025/04/04 15:43:56.502 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:43:56.502 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:43:56.645 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=2891278e-8995-4bc9-a51c-1c8d2d8edcce] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:43:56.795 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=6dd1b716-6705-4cb8-b5b1-7a132e98bf99] [processorDetail="{\"table_ids\":[]}"] [Pipeline] // stage [Pipeline] } [2025/04/04 15:43:56.945 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=f2ee8e9d-25ce-4230-8aba-e199601192fe] [processorDetail="{\"table_ids\":[114]}"] [2025/04/04 15:43:56.945 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"2891278e-8995-4bc9-a51c-1c8d2d8edcce\":[],\"6dd1b716-6705-4cb8-b5b1-7a132e98bf99\":[],\"f2ee8e9d-25ce-4230-8aba-e199601192fe\":[{\"ID\":114,\"Changefeed\":\"b4851339-3cfb-4f87-b4fb-bd6f153419fd\"}]}"] [2025/04/04 15:43:56.945 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=b4851339-3cfb-4f87-b4fb-bd6f153419fd&target-cp-id=6dd1b716-6705-4cb8-b5b1-7a132e98bf99&table-id=114"] [2025/04/04 15:43:56.995 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=114] [2025/04/04 15:43:56.995 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=f2ee8e9d-25ce-4230-8aba-e199601192fe] [targetCapture=6dd1b716-6705-4cb8-b5b1-7a132e98bf99] table move_table.check1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360521540126041 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06a4e4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:15201, start at 2025-04-04 15:43:56.746238717 +0800 CST m=+1.464975658 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:56.755 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:56.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:56.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360521540126041 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06a4e4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:15201, start at 2025-04-04 15:43:56.746238717 +0800 CST m=+1.464975658 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:56.755 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:56.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:56.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360530516082684 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06a72c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:15290, start at 2025-04-04 15:43:56.919137037 +0800 CST m=+1.553705742 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:56.927 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:56.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:56.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table sink_hang.t2 exists check diff failed 1-th time, retry later table move_table.check1 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035de750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114291662749699 MinTableBarrierTs:457114291924893700 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/794c2845-c90d-4abe-a0a4-57e4d893eccf {"id":"794c2845-c90d-4abe-a0a4-57e4d893eccf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13246ca 794c2845-c90d-4abe-a0a4-57e4d893eccf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114291662749699,"min-table-barrier-ts":457114291924893700,"admin-job-type":0} /tidb/cdc/default/default/task/position/794c2845-c90d-4abe-a0a4-57e4d893eccf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035de750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114291662749699 MinTableBarrierTs:457114291924893700 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/794c2845-c90d-4abe-a0a4-57e4d893eccf {"id":"794c2845-c90d-4abe-a0a4-57e4d893eccf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13246ca 794c2845-c90d-4abe-a0a4-57e4d893eccf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114291662749699,"min-table-barrier-ts":457114291924893700,"admin-job-type":0} /tidb/cdc/default/default/task/position/794c2845-c90d-4abe-a0a4-57e4d893eccf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035de750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114291662749699 MinTableBarrierTs:457114291924893700 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114291924893700, checkpointTs: 457114291924893700, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114291924893698, checkpointTs: 457114291924893698, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/794c2845-c90d-4abe-a0a4-57e4d893eccf {"id":"794c2845-c90d-4abe-a0a4-57e4d893eccf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13246ca 794c2845-c90d-4abe-a0a4-57e4d893eccf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114291662749699,"min-table-barrier-ts":457114291924893700,"admin-job-type":0} /tidb/cdc/default/default/task/position/794c2845-c90d-4abe-a0a4-57e4d893eccf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.16733.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_ts_not_forward ddl-only-block-related-table check diff failed 2-th time, retry later table move_table.check1 not exists for 3-th check, retry later + set +x + tso='457114292054917121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114292054917121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:01 CST 2025] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1676516767.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308457274525 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039a280002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10869, start at 2025-04-04 15:43:06.891007688 +0800 CST m=+1.451218716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... table move_table.check1 not exists for 4-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:04 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3445aaed-cc64-47ce-b366-3078771c9c99 {"id":"3445aaed-cc64-47ce-b366-3078771c9c99","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752641} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc18d77dc 3445aaed-cc64-47ce-b366-3078771c9c99 /tidb/cdc/default/default/upstream/7489360521540126041 {"id":7489360521540126041,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3445aaed-cc64-47ce-b366-3078771c9c99 {"id":"3445aaed-cc64-47ce-b366-3078771c9c99","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752641} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc18d77dc 3445aaed-cc64-47ce-b366-3078771c9c99 /tidb/cdc/default/default/upstream/7489360521540126041 {"id":7489360521540126041,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3445aaed-cc64-47ce-b366-3078771c9c99 {"id":"3445aaed-cc64-47ce-b366-3078771c9c99","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752641} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc18d77dc 3445aaed-cc64-47ce-b366-3078771c9c99 /tidb/cdc/default/default/upstream/7489360521540126041 {"id":7489360521540126041,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:44:04 CST 2025] <<<<<< START pulsar cluster in savepoint case >>>>>>' [Fri Apr 4 15:44:04 CST 2025] <<<<<< START pulsar cluster in savepoint case >>>>>> + workdir=/tmp/tidb_cdc_test/savepoint + cluster_type=normal + cd /tmp/tidb_cdc_test/savepoint + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/savepoint/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/savepoint/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/savepoint/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/savepoint/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/savepoint/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/savepoint/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/savepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/savepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/savepoint/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:44:04 CST 2025] <<<<<< START pulsar cluster in normal mode in savepoint case >>>>>>' [Fri Apr 4 15:44:04 CST 2025] <<<<<< START pulsar cluster in normal mode in savepoint case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/savepoint/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/savepoint/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/savepoint/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:04 CST 2025] <<<<<< run test case sink_hang success! >>>>>> table move_table.check1 not exists for 5-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:43:52.973","puller_resolved_ts":"2025-04-04 15:43:45.973","last_synced_ts":"2025-04-04 15:41:37.023","now_ts":"2025-04-04 15:43:54.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:43:52.973","puller_resolved_ts":"2025-04-04' '15:43:45.973","last_synced_ts":"2025-04-04' '15:41:37.023","now_ts":"2025-04-04' '15:43:54.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 11201 7.8 0.0 14968772 146032 ? Sl 15:41 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 11269 5.8 0.0 14967748 144012 ? Sl 15:41 0:09 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":"2025-04-04T15:44:00.845785+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:00.84589+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:44:00.850848+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:00.850911+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:44:00.857335+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:00.861165+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:00.920278+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:00.920341+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:44:00.931755+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1743752642.4111187,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0056268c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752642.411169,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752642.4392438,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022648c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752642.4393003,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752642.5247295,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002492a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752642.524772,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} check_ts_not_forward ddl-only-block-related-table + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 {"level":"warn","ts":"2025-04-04T15:44:06.85834+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:06.862428+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:06.932902+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table move_table.check1 not exists for 6-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table move_table.check1 not exists for 7-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:44:10 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:44:10 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.17231.out cli changefeed create --start-ts=457114292054917121 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-savepoint-test-16575?protocol=canal-json&enable-tidb-extension=true' table move_table.check1 not exists for 8-th check, retry later Create changefeed successfully! ID: 9574b5af-5da2-4fa3-aeed-524879892a57 Info: {"upstream_id":7489360521540126041,"namespace":"default","id":"9574b5af-5da2-4fa3-aeed-524879892a57","sink_uri":"pulsar://127.0.0.1:6650/ticdc-savepoint-test-16575?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:11.775839499+08:00","start_ts":457114292054917121,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114292054917121,"checkpoint_ts":457114292054917121,"checkpoint_time":"2025-04-04 15:43:59.980"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... {"level":"warn","ts":1743752651.4968457,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0056268c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752651.524191,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022648c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:12.859839+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:12.862786+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:12.933491+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + set +x [Fri Apr 4 15:44:13 CST 2025] <<<<<< START Pulsar consumer in savepoint case >>>>>> table move_table.check1 not exists for 9-th check, retry later table savepoint.finish_mark not exists for 1-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } table move_table.check1 not exists for 10-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [Pipeline] // podTemplate [Pipeline] } table savepoint.finish_mark not exists for 2-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 table move_table.check1 exists check diff successfully table savepoint.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 wait process cdc.test exit for 2-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:18.860657+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:18.864841+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:18.934362+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:19 CST 2025] <<<<<< run test case savepoint success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360313431937432 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039968000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10792, start at 2025-04-04 15:43:06.869277147 +0800 CST m=+1.501707644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.877 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.842 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.842 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308457274525 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039a280002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-bmrs-p4mj3, pid:10869, start at 2025-04-04 15:43:06.891007688 +0800 CST m=+1.451218716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.898 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.16616.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:44:21.650 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:44:21.653 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:44:21.654 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=f2ee8e9d-25ce-4230-8aba-e199601192fe] [2025/04/04 15:44:21.654 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:44:21.655 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:44:21.794 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=2891278e-8995-4bc9-a51c-1c8d2d8edcce] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:44:21.945 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=6dd1b716-6705-4cb8-b5b1-7a132e98bf99] [processorDetail="{\"table_ids\":[116]}"] [2025/04/04 15:44:22.094 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=b4851339-3cfb-4f87-b4fb-bd6f153419fd] [captureID=f2ee8e9d-25ce-4230-8aba-e199601192fe] [processorDetail="{\"table_ids\":[118]}"] [2025/04/04 15:44:22.094 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"2891278e-8995-4bc9-a51c-1c8d2d8edcce\":[],\"6dd1b716-6705-4cb8-b5b1-7a132e98bf99\":[{\"ID\":116,\"Changefeed\":\"b4851339-3cfb-4f87-b4fb-bd6f153419fd\"}],\"f2ee8e9d-25ce-4230-8aba-e199601192fe\":[{\"ID\":118,\"Changefeed\":\"b4851339-3cfb-4f87-b4fb-bd6f153419fd\"}]}"] [2025/04/04 15:44:22.094 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=b4851339-3cfb-4f87-b4fb-bd6f153419fd&target-cp-id=f2ee8e9d-25ce-4230-8aba-e199601192fe&table-id=116"] [2025/04/04 15:44:22.145 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:44:22.145 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=6dd1b716-6705-4cb8-b5b1-7a132e98bf99] [targetCapture=f2ee8e9d-25ce-4230-8aba-e199601192fe] check diff successfully 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 table move_table.check2 exists check diff successfully 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 wait process cdc.test exit for 1-th time... + set +x + tso='457114297822871553 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114297822871553 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114297822871553 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:44:23 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1665316655.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 wait process cdc.test exit for 3-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:24.861347+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:24.865707+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:24.93496+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:44:25 CST 2025] <<<<<< run test case move_table success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360313431937432 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:43:15.081245786 +0800 CST StartTs:457114278989922305 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004328750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114280274690055} {CheckpointTs:457114280628584453 MinTableBarrierTs:457114280628584453 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06400859-7c3e-4b6a-b296-c1e4004bdf05 {"id":"06400859-7c3e-4b6a-b296-c1e4004bdf05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752663} /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1e3ccd6 06400859-7c3e-4b6a-b296-c1e4004bdf05 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360313431937432,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:15.081245786+08:00","start-ts":457114278989922305,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114280274690055} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114280628584453,"min-table-barrier-ts":457114280628584453,"admin-job-type":0} /tidb/cdc/default/default/task/position/06400859-7c3e-4b6a-b296-c1e4004bdf05/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a9a5038e-f1eb-4c14-a149-b51e0d6ec137/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360313431937432 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:43:15.081245786 +0800 CST StartTs:457114278989922305 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004328750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114280274690055} {CheckpointTs:457114280628584453 MinTableBarrierTs:457114280628584453 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06400859-7c3e-4b6a-b296-c1e4004bdf05 {"id":"06400859-7c3e-4b6a-b296-c1e4004bdf05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752663} /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1e3ccd6 06400859-7c3e-4b6a-b296-c1e4004bdf05 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360313431937432,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:15.081245786+08:00","start-ts":457114278989922305,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114280274690055} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114280628584453,"min-table-barrier-ts":457114280628584453,"admin-job-type":0} /tidb/cdc/default/default/task/position/06400859-7c3e-4b6a-b296-c1e4004bdf05/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a9a5038e-f1eb-4c14-a149-b51e0d6ec137/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489360313431937432 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:43:15.081245786 +0800 CST StartTs:457114278989922305 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004328750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114280274690055} {CheckpointTs:457114280628584453 MinTableBarrierTs:457114280628584453 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06400859-7c3e-4b6a-b296-c1e4004bdf05 {"id":"06400859-7c3e-4b6a-b296-c1e4004bdf05","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752663} /tidb/cdc/default/__cdc_meta__/capture/a9a5038e-f1eb-4c14-a149-b51e0d6ec137 {"id":"a9a5038e-f1eb-4c14-a149-b51e0d6ec137","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ce80d9 a9a5038e-f1eb-4c14-a149-b51e0d6ec137 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1e3ccd6 06400859-7c3e-4b6a-b296-c1e4004bdf05 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489360313431937432,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:43:15.081245786+08:00","start-ts":457114278989922305,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114280274690055} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114280628584453,"min-table-barrier-ts":457114280628584453,"admin-job-type":0} /tidb/cdc/default/default/task/position/06400859-7c3e-4b6a-b296-c1e4004bdf05/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a9a5038e-f1eb-4c14-a149-b51e0d6ec137/test-1 {"checkpoint-ts":0,"resolved-t+ grep -q 'etcd info' s":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360313431937432 {"id":7489360313431937432,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114297822871553 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.16712.out cli changefeed create --start-ts=457114297822871553 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489360313431937432,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:28.333638548+08:00","start_ts":457114297822871553,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114297822871553,"checkpoint_ts":457114297822871553,"checkpoint_time":"2025-04-04 15:44:21.983"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:30.862425+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:30.867021+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:30.935842+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 Verifying downstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:35.846838+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:35.846882+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:44:35.85279+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:35.852838+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:44:35.921358+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:44:35.921404+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:36.864303+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:36.867837+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:36.93684+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":1743752677.4128695,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0056268c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752677.4129179,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752677.4409683,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022648c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752677.4410136,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752677.5267878,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002492a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752677.5269017,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360699605329640 TiDB Cluster ID. tikv_gc_leader_uuid 657ff092b48000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:18461, start at 2025-04-04 15:44:38.114642952 +0800 CST m=+1.554139162 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:38.122 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:38.098 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:38.098 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360699605329640 TiDB Cluster ID. tikv_gc_leader_uuid 657ff092b48000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:18461, start at 2025-04-04 15:44:38.114642952 +0800 CST m=+1.554139162 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:38.122 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:38.098 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:38.098 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360705760146191 TiDB Cluster ID. tikv_gc_leader_uuid 657ff092b60000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:18541, start at 2025-04-04 15:44:38.147853194 +0800 CST m=+1.517174846 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:38.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:38.154 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:38.154 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 Verifying downstream PD is started... + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19922.out cli tso query --pd=http://127.0.0.1:2379 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 + set +x + tso='457114302873337857 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114302873337857 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114302873337857 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:44:42 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1995519957.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:44:42.865075+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e20a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:42.868633+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001301c00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:44:42.937677+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024f8fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Starting Upstream TiDB... 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 35 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:45 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/005d6a43-4cb2-40c3-8bc0-aaf69e8451ad {"id":"005d6a43-4cb2-40c3-8bc0-aaf69e8451ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752683} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23295d8 005d6a43-4cb2-40c3-8bc0-aaf69e8451ad /tidb/cdc/default/default/upstream/7489360699605329640 {"id":7489360699605329640,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/005d6a43-4cb2-40c3-8bc0-aaf69e8451ad {"id":"005d6a43-4cb2-40c3-8bc0-aaf69e8451ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752683} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23295d8 005d6a43-4cb2-40c3-8bc0-aaf69e8451ad /tidb/cdc/default/default/upstream/7489360699605329640 {"id":7489360699605329640,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/005d6a43-4cb2-40c3-8bc0-aaf69e8451ad {"id":"005d6a43-4cb2-40c3-8bc0-aaf69e8451ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752683} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc23295d8 005d6a43-4cb2-40c3-8bc0-aaf69e8451ad /tidb/cdc/default/default/upstream/7489360699605329640 {"id":7489360699605329640,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114302873337857 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.20015.out cli changefeed create --start-ts=457114302873337857 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360699605329640,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:46.249018919+08:00","start_ts":457114302873337857,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114302873337857,"checkpoint_ts":457114302873337857,"checkpoint_time":"2025-04-04 15:44:41.249"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360731145396080 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09a5f4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:32200, start at 2025-04-04 15:44:45.982949592 +0800 CST m=+1.576139221 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:45.991 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:45.949 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:45.949 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360731145396080 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09a5f4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:32200, start at 2025-04-04 15:44:45.982949592 +0800 CST m=+1.576139221 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:45.991 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:45.949 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:45.949 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360738620157287 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09a7700007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:32282, start at 2025-04-04 15:44:46.056405986 +0800 CST m=+1.572401937 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:46.067 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:46.044 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:46.044 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 5390 0 --:--:-- --:--:-- --:--:-- 5390 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:44:47.599","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:44:47.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:47.599","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:47.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:47.599","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:47.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:44:47.599' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:47.599","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:47.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:44:47.599","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:44:47.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:44:47' + echo 'sink_checkpoint_ts is 2025-04-04' 15:44:47.599 sink_checkpoint_ts is 2025-04-04 15:44:47.599 ++ date -d '2025-04-04 15:44:47.599' +%s + checkpoint_timestamp=1743752687 ++ date -d '2025-04-04 15:44:47' +%s + current_timestamp=1743752687 + '[' 0 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [Fri Apr 4 15:44:48 CST 2025] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.3372233724.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t1 exists + sleep 5 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:51 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f55d61c6-1c22-43b1-beaa-da3e9f07e72f {"id":"f55d61c6-1c22-43b1-beaa-da3e9f07e72f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24d58d4 f55d61c6-1c22-43b1-beaa-da3e9f07e72f /tidb/cdc/default/default/upstream/7489360731145396080 {"id":7489360731145396080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f55d61c6-1c22-43b1-beaa-da3e9f07e72f {"id":"f55d61c6-1c22-43b1-beaa-da3e9f07e72f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24d58d4 f55d61c6-1c22-43b1-beaa-da3e9f07e72f /tidb/cdc/default/default/upstream/7489360731145396080 {"id":7489360731145396080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f55d61c6-1c22-43b1-beaa-da3e9f07e72f {"id":"f55d61c6-1c22-43b1-beaa-da3e9f07e72f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752689} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24d58d4 f55d61c6-1c22-43b1-beaa-da3e9f07e72f /tidb/cdc/default/default/upstream/7489360731145396080 {"id":7489360731145396080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:44:51 CST 2025] <<<<<< START pulsar cluster in autorandom case >>>>>>' [Fri Apr 4 15:44:51 CST 2025] <<<<<< START pulsar cluster in autorandom case >>>>>> + workdir=/tmp/tidb_cdc_test/autorandom + cluster_type=normal + cd /tmp/tidb_cdc_test/autorandom + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/autorandom/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/autorandom/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/autorandom/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/autorandom/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/autorandom/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/autorandom/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/autorandom/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/autorandom/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/autorandom/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:44:51 CST 2025] <<<<<< START pulsar cluster in normal mode in autorandom case >>>>>>' [Fri Apr 4 15:44:51 CST 2025] <<<<<< START pulsar cluster in normal mode in autorandom case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/autorandom/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/autorandom/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/autorandom/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 pass check, checkpoint tso not forward after 10s run task successfully wait process 10256 exit for 1-th time... + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 wait process 10256 exit for 2-th time... wait process 10256 exit for 3-th time... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3155 0 --:--:-- --:--:-- --:--:-- 3155 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:44:54.599","puller_resolved_ts":"2025-04-04 15:44:47.799","last_synced_ts":"2025-04-04 15:44:47.799","now_ts":"2025-04-04 15:44:54.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:54.599","puller_resolved_ts":"2025-04-04' '15:44:47.799","last_synced_ts":"2025-04-04' '15:44:47.799","now_ts":"2025-04-04' '15:44:54.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:54.599","puller_resolved_ts":"2025-04-04' '15:44:47.799","last_synced_ts":"2025-04-04' '15:44:47.799","now_ts":"2025-04-04' '15:44:54.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 wait process 10256 exit for 4-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (10256) - No such process wait process 10256 exit for 5-th time... process 10256 already exit [Fri Apr 4 15:44:55 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.1084910851.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... public/default ++ date + echo '[Fri Apr 4 15:44:58 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:44:58 CST 2025] <<<<<< pulsar is ready >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:58 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004cac000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114292029751324 MinTableBarrierTs:457114307142352902 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc003009-809b-43c4-b9cd-e56062bc47cf {"id":"bc003009-809b-43c4-b9cd-e56062bc47cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752695} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13248bd bc003009-809b-43c4-b9cd-e56062bc47cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114292029751324,"min-table-barrier-ts":457114307142352902,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc003009-809b-43c4-b9cd-e56062bc47cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004cac000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114292029751324 MinTableBarrierTs:457114307142352902 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc003009-809b-43c4-b9cd-e56062bc47cf {"id":"bc003009-809b-43c4-b9cd-e56062bc47cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752695} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13248bd bc003009-809b-43c4-b9cd-e56062bc47cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114292029751324,"min-table-barrier-ts":457114307142352902,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc003009-809b-43c4-b9cd-e56062bc47cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489360422517814101 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json&enable-tidb-extension=true CreateTime:2025-04-04 15:43:45.162195376 +0800 CST StartTs:457114288018948098 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004cac000 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ge8bfbe4ba Epoch:457114288163389447} {CheckpointTs:457114292029751324 MinTableBarrierTs:457114307142352902 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114292029751324, checkpointTs: 457114292029751324, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc003009-809b-43c4-b9cd-e56062bc47cf {"id":"bc003009-809b-43c4-b9cd-e56062bc47cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752695} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc13248bd bc003009-809b-43c4-b9cd-e56062bc47cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489360422517814101,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-7370?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2025-04-04T15:43:45.162195376+08:00","start-ts":457114288018948098,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ge8bfbe4ba","epoch":457114288163389447} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114292029751324,"min-table-barrier-ts":457114307142352902,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc003009-809b-43c4-b9cd-e56062bc47cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360422517814101 {"id":7489360422517814101,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 1-th time, retry later Create changefeed successfully! ID: 7c9d866e-ea16-47b7-9f96-5f1766c42420 Info: {"upstream_id":7489360731145396080,"namespace":"default","id":"7c9d866e-ea16-47b7-9f96-5f1766c42420","sink_uri":"pulsar://127.0.0.1:6650/ticdc-autorandom-test-31579?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:44:59.001121394+08:00","start_ts":457114307369107461,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114307369107461,"checkpoint_ts":457114307369107461,"checkpoint_time":"2025-04-04 15:44:58.399"} [Fri Apr 4 15:44:59 CST 2025] <<<<<< START Pulsar consumer in autorandom case >>>>>> table autorandom_test.table_a not exists for 1-th check, retry later check diff successfully check_ts_forward ddl-only-block-related-table table autorandom_test.table_a exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 1-th time, retry later changefeed is working normally rts: 457114307929047045->457114308190928902 checkpoint: 457114307929047045->457114308190928902 run task successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... check diff successfully wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:04 CST 2025] <<<<<< run test case ddl_only_block_related_table success! >>>>>> wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360815297438391 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ac1c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:14947, start at 2025-04-04 15:45:04.157480505 +0800 CST m=+1.464271412 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:04.165 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:04.164 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:04.164 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360815297438391 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ac1c8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:14947, start at 2025-04-04 15:45:04.157480505 +0800 CST m=+1.464271412 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:04.165 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:04.164 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:04.164 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360820723659286 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0ac344000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:15009, start at 2025-04-04 15:45:04.243822007 +0800 CST m=+1.488370421 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:04.251 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:04.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:04.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:04 CST 2025] <<<<<< run test case autorandom success! >>>>>> TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16388.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114309732597761 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114309732597761 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114309732597761 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:45:08 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1642716429.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:11 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c6e77ab-1689-4dcd-8c2c-822c09f92379 {"id":"9c6e77ab-1689-4dcd-8c2c-822c09f92379","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc294e1d8 9c6e77ab-1689-4dcd-8c2c-822c09f92379 /tidb/cdc/default/default/upstream/7489360815297438391 {"id":7489360815297438391,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c6e77ab-1689-4dcd-8c2c-822c09f92379 {"id":"9c6e77ab-1689-4dcd-8c2c-822c09f92379","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc294e1d8 9c6e77ab-1689-4dcd-8c2c-822c09f92379 /tidb/cdc/default/default/upstream/7489360815297438391 {"id":7489360815297438391,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c6e77ab-1689-4dcd-8c2c-822c09f92379 {"id":"9c6e77ab-1689-4dcd-8c2c-822c09f92379","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752709} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc294e1d8 9c6e77ab-1689-4dcd-8c2c-822c09f92379 /tidb/cdc/default/default/upstream/7489360815297438391 {"id":7489360815297438391,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114309732597761 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16485.out cli changefeed create --start-ts=457114309732597761 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360815297438391,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:45:12.367664736+08:00","start_ts":457114309732597761,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114309732597761,"checkpoint_ts":457114309732597761,"checkpoint_time":"2025-04-04 15:45:07.415"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.t1 exists + sleep 5 start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 14195 25.1 0.6 5130216 2477052 ? Sl 15:45 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 14196 34.8 0.6 5169128 2552476 ? Sl 15:45 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 14197 25.5 0.6 5128168 2484248 ? Sl 15:45 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 14200 33.6 0.6 5160936 2538064 ? Sl 15:45 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 5926 0 --:--:-- --:--:-- --:--:-- 5785 100 243 100 243 0 0 5926 0 --:--:-- --:--:-- --:--:-- 5785 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:45:19.714","puller_resolved_ts":"2025-04-04 15:45:13.815","last_synced_ts":"2025-04-04 15:45:14.315","now_ts":"2025-04-04 15:45:21.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:19.714","puller_resolved_ts":"2025-04-04' '15:45:13.815","last_synced_ts":"2025-04-04' '15:45:14.315","now_ts":"2025-04-04' '15:45:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:19.714","puller_resolved_ts":"2025-04-04' '15:45:13.815","last_synced_ts":"2025-04-04' '15:45:14.315","now_ts":"2025-04-04' '15:45:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360897306984144 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bee4c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:35521, start at 2025-04-04 15:45:23.376017209 +0800 CST m=+1.496380805 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360897306984144 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bee4c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:35521, start at 2025-04-04 15:45:23.376017209 +0800 CST m=+1.496380805 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360898400384906 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0befbc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-vsw6-5l32b, pid:35587, start at 2025-04-04 15:45:23.468071191 +0800 CST m=+1.521277911 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.478 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.439 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.439 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360897869441762 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bf260000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:12125, start at 2025-04-04 15:45:23.644680728 +0800 CST m=+1.616493998 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.658 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.658 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360897869441762 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bf260000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:12125, start at 2025-04-04 15:45:23.644680728 +0800 CST m=+1.616493998 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.658 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.658 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360900067644666 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0bf3e4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-kpwn-bkjc5, pid:12189, start at 2025-04-04 15:45:23.739799771 +0800 CST m=+1.647674012 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:23.748 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:23.705 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:23.705 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + '[' pulsar == mysql ']' + stop_tidb_cluster + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.36905.out cli tso query --pd=http://127.0.0.1:2379 ***************** properties ***************** "mysql.port"="4000" "requestdistribution"="uniform" "updateproportion"="0" "readallfields"="true" "mysql.user"="root" "mysql.host"="127.0.0.1" "scanproportion"="0" "dotransactions"="false" "recordcount"="20" "threadcount"="4" "readproportion"="0" "operationcount"="0" "workload"="core" "mysql.db"="changefeed_auto_stop_1" "insertproportion"="0" ********************************************** Run finished, takes 11.22338ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3026.8, Avg(us): 2034, Min(us): 1145, Max(us): 4597, 95th(us): 5000, 99th(us): 5000 + set +x + tso='457114314773889025 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114314773889025 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:45:28 CST 2025] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.3694636948.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "readproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.db"="changefeed_auto_stop_2" "dotransactions"="false" "recordcount"="20" "mysql.port"="4000" "mysql.user"="root" "threadcount"="4" "scanproportion"="0" "readallfields"="true" "updateproportion"="0" "insertproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 12.744588ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2742.5, Avg(us): 2391, Min(us): 1170, Max(us): 5390, 95th(us): 6000, 99th(us): 6000 ***************** properties ***************** "mysql.port"="4000" "threadcount"="4" "operationcount"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "requestdistribution"="uniform" "dotransactions"="false" "workload"="core" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_3" "scanproportion"="0" "recordcount"="20" "insertproportion"="0" "readallfields"="true" ********************************************** Run finished, takes 11.137789ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2898.5, Avg(us): 1993, Min(us): 1112, Max(us): 4477, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "updateproportion"="0" "insertproportion"="0" "workload"="core" "mysql.db"="changefeed_auto_stop_4" "scanproportion"="0" "requestdistribution"="uniform" "readallfields"="true" "threadcount"="4" "readproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="20" "mysql.port"="4000" "operationcount"="0" "mysql.user"="root" "dotransactions"="false" ********************************************** Run finished, takes 10.406279ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3290.6, Avg(us): 1904, Min(us): 1009, Max(us): 4280, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:45:30 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1377513777.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:31 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb011ea1-a486-4f25-9ce0-a16925c91cc8 {"id":"cb011ea1-a486-4f25-9ce0-a16925c91cc8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752728} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e7a1d7 cb011ea1-a486-4f25-9ce0-a16925c91cc8 /tidb/cdc/default/default/upstream/7489360897306984144 {"id":7489360897306984144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb011ea1-a486-4f25-9ce0-a16925c91cc8 {"id":"cb011ea1-a486-4f25-9ce0-a16925c91cc8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752728} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e7a1d7 cb011ea1-a486-4f25-9ce0-a16925c91cc8 /tidb/cdc/default/default/upstream/7489360897306984144 {"id":7489360897306984144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb011ea1-a486-4f25-9ce0-a16925c91cc8 {"id":"cb011ea1-a486-4f25-9ce0-a16925c91cc8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752728} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e7a1d7 cb011ea1-a486-4f25-9ce0-a16925c91cc8 /tidb/cdc/default/default/upstream/7489360897306984144 {"id":7489360897306984144,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:45:31 CST 2025] <<<<<< START pulsar cluster in generate_column case >>>>>>' [Fri Apr 4 15:45:31 CST 2025] <<<<<< START pulsar cluster in generate_column case >>>>>> + workdir=/tmp/tidb_cdc_test/generate_column + cluster_type=normal + cd /tmp/tidb_cdc_test/generate_column + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/generate_column/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/generate_column/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/generate_column/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/generate_column/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/generate_column/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/generate_column/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/generate_column/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/generate_column/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/generate_column/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/generate_column/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/generate_column/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/generate_column/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:45:31 CST 2025] <<<<<< START pulsar cluster in normal mode in generate_column case >>>>>>' [Fri Apr 4 15:45:31 CST 2025] <<<<<< START pulsar cluster in normal mode in generate_column case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/generate_column/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/generate_column/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/generate_column/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + stop_tidb_cluster + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:33 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:45:33 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1383213834.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:36 GMT < Content-Length: 1275 < Content-Type: text/plain; charset=utf-8 < { [1275 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/601337b2-fd46-459b-baa6-6b0ee6b13ccc {"id":"601337b2-fd46-459b-baa6-6b0ee6b13ccc","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752734} /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e45967 601337b2-fd46-459b-baa6-6b0ee6b13ccc /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/601337b2-fd46-459b-baa6-6b0ee6b13ccc {"id":"601337b2-fd46-459b-baa6-6b0ee6b13ccc","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752734} /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e45967 601337b2-fd46-459b-baa6-6b0ee6b13ccc /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/601337b2-fd46-459b-baa6-6b0ee6b13ccc {"id":"601337b2-fd46-459b-baa6-6b0ee6b13ccc","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752734} /tidb/cdc/default/__cdc_meta__/capture/d99b627a-176b-4365-9fa6-3257fd94ef4f {"id":"d99b627a-176b-4365-9fa6-3257fd94ef4f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e4592f d99b627a-176b-4365-9fa6-3257fd94ef4f /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e45967 601337b2-fd46-459b-baa6-6b0ee6b13ccc /tidb/cdc/default/default/upstream/7489360897869441762 {"id":7489360897869441762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Fri Apr 4 15:45:36 CST 2025] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>>' [Fri Apr 4 15:45:36 CST 2025] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_auto_stop + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_auto_stop + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Fri Apr 4 15:45:36 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>>' [Fri Apr 4 15:45:36 CST 2025] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 public/default ++ date + echo '[Fri Apr 4 15:45:37 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:45:37 CST 2025] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.37409.out cli changefeed create --start-ts=457114314773889025 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-generate-column-test-16637?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: d68f5c1d-a315-46e5-9b5d-f472c044654c Info: {"upstream_id":7489360897306984144,"namespace":"default","id":"d68f5c1d-a315-46e5-9b5d-f472c044654c","sink_uri":"pulsar://127.0.0.1:6650/ticdc-generate-column-test-16637?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2025-04-04T15:45:38.638406113+08:00","start_ts":457114314773889025,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114314773889025,"checkpoint_ts":457114314773889025,"checkpoint_time":"2025-04-04 15:45:26.646"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + set +x [Fri Apr 4 15:45:40 CST 2025] <<<<<< START Pulsar consumer in generate_column case >>>>>> + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table generate_column.t1 not exists for 1-th check, retry later public/default ++ date + echo '[Fri Apr 4 15:45:43 CST 2025] <<<<<< pulsar is ready >>>>>>' [Fri Apr 4 15:45:43 CST 2025] <<<<<< pulsar is ready >>>>>> table generate_column.t1 exists check diff failed 1-th time, retry later [Fri Apr 4 15:45:43 CST 2025] <<<<<< START Pulsar consumer in changefeed_auto_stop case >>>>>> check_changefeed_state http://127.0.0.1:2379 d8220e14-d0f2-4579-96d5-3073e2db3d0e normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=d8220e14-d0f2-4579-96d5-3073e2db3d0e + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c d8220e14-d0f2-4579-96d5-3073e2db3d0e -s + info='{ "upstream_id": 7489360897869441762, "namespace": "default", "id": "d8220e14-d0f2-4579-96d5-3073e2db3d0e", "state": "normal", "checkpoint_tso": 457114314724605953, "checkpoint_time": "2025-04-04 15:45:26.458", "error": null }' + echo '{ "upstream_id": 7489360897869441762, "namespace": "default", "id": "d8220e14-d0f2-4579-96d5-3073e2db3d0e", "state": "normal", "checkpoint_tso": 457114314724605953, "checkpoint_time": "2025-04-04 15:45:26.458", "error": null }' { "upstream_id": 7489360897869441762, "namespace": "default", "id": "d8220e14-d0f2-4579-96d5-3073e2db3d0e", "state": "normal", "checkpoint_tso": 457114314724605953, "checkpoint_time": "2025-04-04 15:45:26.458", "error": null } ++ echo '{' '"upstream_id":' 7489360897869441762, '"namespace":' '"default",' '"id":' '"d8220e14-d0f2-4579-96d5-3073e2db3d0e",' '"state":' '"normal",' '"checkpoint_tso":' 457114314724605953, '"checkpoint_time":' '"2025-04-04' '15:45:26.458",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360897869441762, '"namespace":' '"default",' '"id":' '"d8220e14-d0f2-4579-96d5-3073e2db3d0e",' '"state":' '"normal",' '"checkpoint_tso":' 457114314724605953, '"checkpoint_time":' '"2025-04-04' '15:45:26.458",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later check diff successfully table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:46 CST 2025] <<<<<< run test case generate_column success! >>>>>> table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:55 CST 2025] <<<<<< run test case changefeed_auto_stop success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6906 0 --:--:-- --:--:-- --:--:-- 6906 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:47:03.748","puller_resolved_ts":"2025-04-04 15:46:56.749","last_synced_ts":"2025-04-04 15:44:47.799","now_ts":"2025-04-04 15:47:04.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:47:03.748","puller_resolved_ts":"2025-04-04' '15:46:56.749","last_synced_ts":"2025-04-04' '15:44:47.799","now_ts":"2025-04-04' '15:47:04.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17605 8.0 0.0 14968512 147388 ? Sl 15:44 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 17673 6.1 0.0 14968256 141792 ? Sl 15:44 0:09 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":"2025-04-04T15:47:11.617984+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:11.618038+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:11.627237+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:11.634973+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:11.635025+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:11.642368+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:11.683509+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:11.683558+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:11.696826+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1743752833.1607952,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b78540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752833.160842,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752833.189096,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0058ba540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752833.1891499,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752833.2476978,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00110c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752833.2477376,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:17.628169+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:17.643257+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:17.697973+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1743752842.2453878,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b78540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752842.278728,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0058ba540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:23.629896+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:23.644291+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:23.698693+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:29.630303+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:29.644782+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:29.700543+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:35.632038+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:35.6466+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:35.701178+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:41.63234+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:41.648501+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:41.702185+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 26777 0 --:--:-- --:--:-- --:--:-- 26777 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:45:20.715","puller_resolved_ts":"2025-04-04 15:45:20.715","last_synced_ts":"2025-04-04 15:45:14.315","now_ts":"2025-04-04 15:47:31.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:20.715","puller_resolved_ts":"2025-04-04' '15:45:20.715","last_synced_ts":"2025-04-04' '15:45:14.315","now_ts":"2025-04-04' '15:47:31.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:45:20.715","puller_resolved_ts":"2025-04-04' '15:45:20.715","last_synced_ts":"2025-04-04' '15:45:14.315","now_ts":"2025-04-04' '15:47:31.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:46.61905+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:46.619095+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:46.635441+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:46.635486+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:47:46.683996+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:47:46.684029+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:47.633893+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release {"level":"warn","ts":"2025-04-04T15:47:47.649917+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:47.70337+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1743752868.162498,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001b78540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752868.1625395,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752868.1902976,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0058ba540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752868.190332,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":1743752868.248884,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00110c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752868.2489264,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361532737614853 TiDB Cluster ID. tikv_gc_leader_uuid 657ff14f0b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:17690, start at 2025-04-04 15:47:50.984284811 +0800 CST m=+1.455316769 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:50.992 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:50.958 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:50.958 +0800 All versions after safe point can be accessed. (DO NOT EDIT) 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361532737614853 TiDB Cluster ID. tikv_gc_leader_uuid 657ff14f0b8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:17690, start at 2025-04-04 15:47:50.984284811 +0800 CST m=+1.455316769 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:50.992 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:50.958 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:50.958 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361530956260426 TiDB Cluster ID. tikv_gc_leader_uuid 657ff14f2fc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:17764, start at 2025-04-04 15:47:51.130544454 +0800 CST m=+1.533513374 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:51.138 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:51.103 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:51.103 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:47:53.635534+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001485a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:53.651733+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002004e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:47:53.705004+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002284c40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19232.out cli tso query --pd=http://127.0.0.1:2379 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 35 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... + set +x + tso='457114353469227009 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114353469227009 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114353469227009 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:47:55 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1926719269.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:58 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 {"id":"4ed9444f-b5db-4ea8-a3bb-69179bc75ea3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752876} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc52291d8 4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 /tidb/cdc/default/default/upstream/7489361532737614853 {"id":7489361532737614853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 {"id":"4ed9444f-b5db-4ea8-a3bb-69179bc75ea3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752876} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc52291d8 4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 /tidb/cdc/default/default/upstream/7489361532737614853 {"id":7489361532737614853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 {"id":"4ed9444f-b5db-4ea8-a3bb-69179bc75ea3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752876} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc52291d8 4ed9444f-b5db-4ea8-a3bb-69179bc75ea3 /tidb/cdc/default/default/upstream/7489361532737614853 {"id":7489361532737614853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114353469227009 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19329.out cli changefeed create --start-ts=457114353469227009 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361532737614853,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:47:59.214776188+08:00","start_ts":457114353469227009,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114353469227009,"checkpoint_ts":457114353469227009,"checkpoint_time":"2025-04-04 15:47:54.257"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 17690 10.5 0.0 3332580 217196 ? Sl 15:47 0:01 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752869516755237.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 17699 4.5 0.0 3258528 190260 ? Sl 15:47 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752869525748043.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 17764 11.7 0.0 3399048 233276 ? Sl 15:47 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752869585411402.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 12789 0 --:--:-- --:--:-- --:--:-- 12789 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:48:04.558","puller_resolved_ts":"2025-04-04 15:48:00.657","last_synced_ts":"2025-04-04 15:48:00.657","now_ts":"2025-04-04 15:48:07.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:04.558","puller_resolved_ts":"2025-04-04' '15:48:00.657","last_synced_ts":"2025-04-04' '15:48:00.657","now_ts":"2025-04-04' '15:48:07.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:04.558","puller_resolved_ts":"2025-04-04' '15:48:00.657","last_synced_ts":"2025-04-04' '15:48:00.657","now_ts":"2025-04-04' '15:48:07.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361628501334170 TiDB Cluster ID. tikv_gc_leader_uuid 657ff165df8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:21344, start at 2025-04-04 15:48:14.357847775 +0800 CST m=+1.470147489 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:14.365 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:14.334 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:14.334 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361628501334170 TiDB Cluster ID. tikv_gc_leader_uuid 657ff165df8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:21344, start at 2025-04-04 15:48:14.357847775 +0800 CST m=+1.470147489 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:14.365 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:14.334 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:14.334 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361632264508843 TiDB Cluster ID. tikv_gc_leader_uuid 657ff16603c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:21425, start at 2025-04-04 15:48:14.514752887 +0800 CST m=+1.544962124 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:50:14.524 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:48:14.529 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:38:14.529 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22815.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114359597367297 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114359597367297 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114359597367297 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:48:19 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2285322855.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:48:22 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfd8434d-6d7e-432f-a896-143b5c18ec56 {"id":"cfd8434d-6d7e-432f-a896-143b5c18ec56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752899} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc57bbcdb cfd8434d-6d7e-432f-a896-143b5c18ec56 /tidb/cdc/default/default/upstream/7489361628501334170 {"id":7489361628501334170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfd8434d-6d7e-432f-a896-143b5c18ec56 {"id":"cfd8434d-6d7e-432f-a896-143b5c18ec56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752899} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc57bbcdb cfd8434d-6d7e-432f-a896-143b5c18ec56 /tidb/cdc/default/default/upstream/7489361628501334170 {"id":7489361628501334170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfd8434d-6d7e-432f-a896-143b5c18ec56 {"id":"cfd8434d-6d7e-432f-a896-143b5c18ec56","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743752899} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc57bbcdb cfd8434d-6d7e-432f-a896-143b5c18ec56 /tidb/cdc/default/default/upstream/7489361628501334170 {"id":7489361628501334170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114359597367297 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22905.out cli changefeed create --start-ts=457114359597367297 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361628501334170,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:48:22.523416106+08:00","start_ts":457114359597367297,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114359597367297,"checkpoint_ts":457114359597367297,"checkpoint_time":"2025-04-04 15:48:17.634"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20615 34.1 0.6 5175776 2553456 ? Sl 15:48 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 20616 26.4 0.6 5152224 2516232 ? Sl 15:48 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 20617 24.7 0.6 5128160 2478616 ? Sl 15:48 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 20626 34.0 0.6 5159904 2538700 ? Sl 15:48 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 6075 0 --:--:-- --:--:-- --:--:-- 6075 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:48:29.934","puller_resolved_ts":"2025-04-04 15:48:23.984","last_synced_ts":"2025-04-04 15:48:23.984","now_ts":"2025-04-04 15:48:31.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:29.934","puller_resolved_ts":"2025-04-04' '15:48:23.984","last_synced_ts":"2025-04-04' '15:48:23.984","now_ts":"2025-04-04' '15:48:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:29.934","puller_resolved_ts":"2025-04-04' '15:48:23.984","last_synced_ts":"2025-04-04' '15:48:23.984","now_ts":"2025-04-04' '15:48:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7620 0 --:--:-- --:--:-- --:--:-- 7892 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:50:14.707","puller_resolved_ts":"2025-04-04 15:50:09.708","last_synced_ts":"2025-04-04 15:48:00.657","now_ts":"2025-04-04 15:50:17.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:50:14.707","puller_resolved_ts":"2025-04-04' '15:50:09.708","last_synced_ts":"2025-04-04' '15:48:00.657","now_ts":"2025-04-04' '15:50:17.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:50:14.707","puller_resolved_ts":"2025-04-04' '15:50:09.708","last_synced_ts":"2025-04-04' '15:48:00.657","now_ts":"2025-04-04' '15:50:17.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362243671623790 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1f172c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:20586, start at 2025-04-04 15:50:37.28956463 +0800 CST m=+1.465817727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:37.297 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:37.259 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:37.259 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362243671623790 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1f172c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:20586, start at 2025-04-04 15:50:37.28956463 +0800 CST m=+1.465817727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:37.297 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:37.259 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:37.259 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362245789960988 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1f18bc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-fl57-46nx8, pid:20655, start at 2025-04-04 15:50:37.38084561 +0800 CST m=+1.498094726 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:52:37.389 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:50:37.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:40:37.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.22051.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114397064036353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114397064036353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114397064036353 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:50:41 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.2208622088.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:50:45 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32ac2ccd-733d-4aa6-bec5-654a4ea63ade {"id":"32ac2ccd-733d-4aa6-bec5-654a4ea63ade","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753042} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7abfcda 32ac2ccd-733d-4aa6-bec5-654a4ea63ade /tidb/cdc/default/default/upstream/7489362243671623790 {"id":7489362243671623790,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32ac2ccd-733d-4aa6-bec5-654a4ea63ade {"id":"32ac2ccd-733d-4aa6-bec5-654a4ea63ade","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753042} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7abfcda 32ac2ccd-733d-4aa6-bec5-654a4ea63ade /tidb/cdc/default/default/upstream/7489362243671623790 {"id":7489362243671623790,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32ac2ccd-733d-4aa6-bec5-654a4ea63ade {"id":"32ac2ccd-733d-4aa6-bec5-654a4ea63ade","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753042} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc7abfcda 32ac2ccd-733d-4aa6-bec5-654a4ea63ade /tidb/cdc/default/default/upstream/7489362243671623790 {"id":7489362243671623790,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114397064036353 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.22145.out cli changefeed create --start-ts=457114397064036353 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362243671623790,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:50:45.435856114+08:00","start_ts":457114397064036353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114397064036353,"checkpoint_ts":457114397064036353,"checkpoint_time":"2025-04-04 15:50:40.558"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 21909 0 --:--:-- --:--:-- --:--:-- 21909 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:48:30.934","puller_resolved_ts":"2025-04-04 15:48:30.934","last_synced_ts":"2025-04-04 15:48:23.984","now_ts":"2025-04-04 15:50:41.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:30.934","puller_resolved_ts":"2025-04-04' '15:48:30.934","last_synced_ts":"2025-04-04' '15:48:23.984","now_ts":"2025-04-04' '15:50:41.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:48:30.934","puller_resolved_ts":"2025-04-04' '15:48:30.934","last_synced_ts":"2025-04-04' '15:48:23.984","now_ts":"2025-04-04' '15:50:41.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362346295505187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff207e080012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:24196, start at 2025-04-04 15:51:00.271543494 +0800 CST m=+1.594955326 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:00.279 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:00.276 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:00.276 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362346295505187 TiDB Cluster ID. tikv_gc_leader_uuid 657ff207e080012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:24196, start at 2025-04-04 15:51:00.271543494 +0800 CST m=+1.594955326 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:00.279 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:00.276 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:00.276 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362341975744112 TiDB Cluster ID. tikv_gc_leader_uuid 657ff207d40000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:24275, start at 2025-04-04 15:51:00.211716769 +0800 CST m=+1.461843058 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:53:00.219 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:51:00.226 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:41:00.226 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.25638.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114403045900289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114403045900289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114403045900289 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:51:04 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2567625678.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:51:07 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfba5e19-ab00-435f-b3e5-4018d156ac6c {"id":"cfba5e19-ab00-435f-b3e5-4018d156ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753065} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc80550d4 cfba5e19-ab00-435f-b3e5-4018d156ac6c /tidb/cdc/default/default/upstream/7489362346295505187 {"id":7489362346295505187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfba5e19-ab00-435f-b3e5-4018d156ac6c {"id":"cfba5e19-ab00-435f-b3e5-4018d156ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753065} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc80550d4 cfba5e19-ab00-435f-b3e5-4018d156ac6c /tidb/cdc/default/default/upstream/7489362346295505187 {"id":7489362346295505187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfba5e19-ab00-435f-b3e5-4018d156ac6c {"id":"cfba5e19-ab00-435f-b3e5-4018d156ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753065} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc80550d4 cfba5e19-ab00-435f-b3e5-4018d156ac6c /tidb/cdc/default/default/upstream/7489362346295505187 {"id":7489362346295505187,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114403045900289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.25729.out cli changefeed create --start-ts=457114403045900289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362346295505187,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:51:08.340737132+08:00","start_ts":457114403045900289,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114403045900289,"checkpoint_ts":457114403045900289,"checkpoint_time":"2025-04-04 15:51:03.377"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 141k 0 --:--:-- --:--:-- --:--:-- 141k + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:50:40.558","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:51:06.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:50:40.558","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:51:06.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:50:40.558","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:51:06.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 24187 4.8 0.0 3479720 189320 ? Sl 15:50 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753058656046970.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 24196 11.3 0.0 3406596 213904 ? Sl 15:50 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753058664734069.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 24275 11.5 0.0 3472776 240404 ? Sl 15:50 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743753058736093674.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 4584 0 --:--:-- --:--:-- --:--:-- 4673 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:51:16.676","puller_resolved_ts":"2025-04-04 15:51:09.776","last_synced_ts":"2025-04-04 15:51:10.276","now_ts":"2025-04-04 15:51:17.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:16.676","puller_resolved_ts":"2025-04-04' '15:51:09.776","last_synced_ts":"2025-04-04' '15:51:10.276","now_ts":"2025-04-04' '15:51:17.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:51:16.676","puller_resolved_ts":"2025-04-04' '15:51:09.776","last_synced_ts":"2025-04-04' '15:51:10.276","now_ts":"2025-04-04' '15:51:17.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Fri Apr 4 15:51:18 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Fri Apr 4 15:51:18 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:53:26.827","puller_resolved_ts":"2025-04-04 15:53:18.827","last_synced_ts":"2025-04-04 15:51:10.276","now_ts":"2025-04-04 15:53:27.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:53:26.827","puller_resolved_ts":"2025-04-04' '15:53:18.827","last_synced_ts":"2025-04-04' '15:51:10.276","now_ts":"2025-04-04' '15:53:27.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:53:26.827","puller_resolved_ts":"2025-04-04' '15:53:18.827","last_synced_ts":"2025-04-04' '15:51:10.276","now_ts":"2025-04-04' '15:53:27.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363054193482756 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2aa2680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:27015, start at 2025-04-04 15:53:46.435148857 +0800 CST m=+1.578883733 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:46.443 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:46.445 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:46.445 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363054193482756 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2aa2680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:27015, start at 2025-04-04 15:53:46.435148857 +0800 CST m=+1.578883733 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:46.443 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:46.445 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:46.445 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489363059391630133 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2aa1a4000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-pulsar-test-3485-mdsr-6978f, pid:27091, start at 2025-04-04 15:53:46.39465668 +0800 CST m=+1.474003536 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:55:46.402 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:53:46.394 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:43:46.394 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28449.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114446605582337 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114446605582337 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114446605582337 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:53:50 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2848528487.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:53:54 GMT < Content-Length: 817 < Content-Type: text/plain; charset=utf-8 < { [817 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8f33197-173a-4b73-b940-c996acd930d2 {"id":"d8f33197-173a-4b73-b940-c996acd930d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8e68d8 d8f33197-173a-4b73-b940-c996acd930d2 /tidb/cdc/default/default/upstream/7489363054193482756 {"id":7489363054193482756,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8f33197-173a-4b73-b940-c996acd930d2 {"id":"d8f33197-173a-4b73-b940-c996acd930d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8e68d8 d8f33197-173a-4b73-b940-c996acd930d2 /tidb/cdc/default/default/upstream/7489363054193482756 {"id":7489363054193482756,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8f33197-173a-4b73-b940-c996acd930d2 {"id":"d8f33197-173a-4b73-b940-c996acd930d2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ge8bfbe4ba","git-hash":"e8bfbe4ba98e05140ab733ee3d08dd3b99918721","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1743753231} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffca8e68d8 d8f33197-173a-4b73-b940-c996acd930d2 /tidb/cdc/default/default/upstream/7489363054193482756 {"id":7489363054193482756,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114446605582337 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28550.out cli changefeed create --start-ts=457114446605582337 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489363054193482756,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:53:54.530305447+08:00","start_ts":457114446605582337,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ge8bfbe4ba","resolved_ts":457114446605582337,"checkpoint_ts":457114446605582337,"checkpoint_time":"2025-04-04 15:53:49.544"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8710 0 --:--:-- --:--:-- --:--:-- 8710 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:53:49.544","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:54:16.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:49.544","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:54:16.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:53:49.544","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:54:16.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 4 15:54:27 CST 2025] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 4 15:54:27 CST 2025] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_pulsar_test-3485/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS