Update Python applications - Managed Service for Apache Flink

Update Python applications

Python applications can use connectors in 2 different ways: packaging connectors and other Java dependencies as part of single uber-jar, or use connector jar directly. To fix applications affected by Async Sink deadlock:

  • If the application uses an uber jar, follow the instructions for Update Java applications.

  • To rebuild connector jars from source, use the following steps:

Building connectors from source:

Prerequisites, similar to Flink build requirements:

  • Java 11

  • Maven 3.2.5

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to kinesis connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-kinesis/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdkv2.version=2.20.144

  5. Navigate to kinesis connector directory

    cd ../flink-sql-connector-kinesis

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-kinesis-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to kinesis connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-aws-kinesis-streams/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdk.version=2.20.144

  5. Navigate to kinesis connector directory

    cd ../flink-sql-connector-aws-kinesis-streams

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-aws-kinesis-streams-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-src.tgz

  2. Uncompress source code:

    tar -xvf flink-1.15.4-src.tgz

  3. Navigate to connector directory

    cd flink-1.15.4/flink-connectors/flink-connector-aws-kinesis-firehose/

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Daws.sdk.version=2.20.144

  5. Navigate to sql connector directory

    cd ../flink-sql-connector-aws-kinesis-firehose

  6. Compile and install sql connector jar:

    mvn clean install -DskipTests -Dfast

  7. Resulting jar will be available at:

    target/flink-sql-connector-aws-kinesis-firehose-1.15.4.jar

  1. Download source code for Flink 1.15.4:

    wget https://archive.apache.org/dist/flink/flink-connector-aws-3.0.0/flink-connector-aws-3.0.0-src.tgz

  2. Uncompress source code:

    tar -xvf flink-connector-aws-3.0.0-src.tgz

  3. Navigate to connector directory

    cd flink-connector-aws-3.0.0

  4. Compile and install connector jar, specifying required AWS SDK version. To speed up build use -DskipTests to skip test execution and -Dfast to skip additional source code checks:

    mvn clean install -DskipTests -Dfast -Dflink.version=1.15.4 -Daws.sdk.version=2.20.144

  5. Resulting jar will be available at:

    flink-sql-connector-dynamodb/target/flink-sql-connector-dynamodb-3.0.0.jar