Best match

Most recent

Wastewater

Texas

LNG spill

An exclusive Inside Climate News analysis found companies have spilled nearly 150 million gallons of toxic, highly saline wastewater in Texas over the last decade.

https://www.bing.com/th?id=OVFT.O2NKcJOJmQjiTtEO2zKNfS&pid=News

Oil and gas companies spill millions of gallons of wastewater in Texas

https://www.bing.com/th?id=ODF.gYpOmzDeSGLiqqB6SR5pCg&pid=news

Texas Standard

The U.S. Energy secretary on Monday announced Arizona is set to receive part of a $1.3 billion federal grant to boost the state's power systems in rural areas.             Speaking at the Apache Generating Station in Cochise County,

Arizona will get some of a billion dollar federal energy grant. Here's why

https://www.bing.com/th?id=ODF.ne24KDFlezOg0DagvpgM2g&pid=news

Arizona Republic on MSN.com

Magnolia

Chord

Permian

Permian Resources ( NYSE: PR ), Chord Energy ( NASDAQ: CHRD ), and Magnolia Oil & Gas ( NYSE: MGY) could be the next acquisition targets in the oil industry as consolidation continues with Exxon Mobil's ( XOM) and Chevron's ( CVX) recent mega transactions.

https://www.bing.com/th?id=OVFT.OCrM9I2CKsKgTLLvRQGzNS&pid=News

Permian Resources, Chord Energy, Magnolia Oil may be next M&A targets - analyst

https://www.bing.com/th?id=ODF.T3WKS_yw6iQ2VstDFu9FMw&pid=news

Seeking Alpha

Hess Corporation

Upstream

Assuming completion of the Chevron deal in the first half of 2024, Hess will join a list of exciting E&P companies that have been acquired in recent times in the US, including BHP Petroleum, Anadarko Petroleum, Kerr-McGee and even Newfield Exploration.

https://www.bing.com/th?id=OVFT.KxxMCxPT8mb7QWrCS4RBnC&pid=News

Farewell to Hess – one of the last ground-breaking E&P companies

https://www.bing.com/th?id=ODF.ZQAYnWwvDleIEPUbJXJbUQ&pid=news

Exxon Mobil

Strong

Wall Street

This company was long considered an industry leader when it was known as Apache, and the stock is perhaps offering ... This company resulted from the $17 billion merger of Cabot Oil & Gas and Cimarex Energy in 2021. Coterra Energy Inc. (NASDAQ: CTRA ...

After Exxon's Huge Pioneer Natural Resources Buy, Could These 5 'Strong Buy' Energy Stocks Be Next?

24/7 Wall St

Donald Trump

Pioneer High School

One of Trump’s biggest oil industry supporters historically has been Hamm, the billionaire and Continental Resources founder who’s donated $769,400 to Trump Victory since 2016. Hamm also played an unofficial role as an energy adviser to Trump on the campaign trail in 2016 and spoke at the 2016 Republican National Convention.

Oil leaders like Anwar, Pioneer group now funding Trump's rivals

https://www.bing.com/th?id=ODF.mzeJPa2LZIiuqSL0c3JwDQ&pid=news

Midland Reporter-Telegram

Mint

Supply chain

India

New Delhi: Hitachi Energy, which helps companies modernize power grids and meet energy transition goals, aims to expand its portfolio and manufacturing capacity in India, and boost its domestic ...

Hitachi Energy aims to build India supply chain: CEO Claudio Facchin

Stock

Price

Energy stocks were climbing in premarket trading Friday as oil prices rose amid worries of further escalation of the conflict in the Middle East. Occidental Petroleum stock (ticker: OXY ...

Energy Stocks Gain as Oil Prices Rise

Barron's

Public utility

The Wall Street Journal

Kruger said the company is making “tangible and materially financial” investments in decarbonizing its existing energy business. Kruger added energy forecasts indicate oil and gas will remain ...

Energy & Utilities Roundup: Market Talk

Wall Street Journal

Eight states and dozens of municipalities have filed lawsuits against oil companies for damages alleged ... they will drive up energy costs for everyone. ”They're basically attempting to ...

Waves of climate lawsuits against oil companies likely to drive up consumer energy costs

KTBS

Get the most recent info and news about Apache on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Apache Corporation is an independent energy company. The Company explores for, develops, and produces natural gas, crude oil, and natural gas liquids. Apache operates worldwide.

Apache

Introduction: Large enterprises, which have more than one business unit, usually have more than one data platform environment. In fact, companies trying to be agile to make quick decisions, all have business unit-specific data platforms. With the computational advancements and rapid growth in data analytics and science, data integration tasks for data platforms are increasing exponentially as they need to support transactional as well as batch processing jobs. Here, the challenge is choosing the orchestration tool, as data integrations have to work in conjunction with different platforms. Though there are several tools in the market, in the article, we will cover how Apache airflow became the most trusted tool for the orchestration industry and its advantages. ETL: Traditional ETL (Extract, Transform, and loading) tools like Informatica, Ab Initio, and Talend all come with their own orchestration process. But as they are proprietary tools, it's expensive as well. Cloud service providers Azure, AWS (Amazon Web Services), and Google Cloud are all trying cash in the orchestration space so Azure has Synapse and AWS has Glue, Step function, etc. Basics: Orchestration is a simple or complex series of workflows defining a process. Workflows depend on one another to perform a business function or a series of business func‐tions calling other services. Then what is the ? process A process is a business function or set of business functions achieved by executing one or more tasks. Then what is the ? task A task is a step in a process. A task may call a service to perform a particular data-related function. Then what is the ? Service A service is software like Azure Synapse Analytics, Python data frame, or AWS DynamoDB that performs tasks. Common Problems in the Legacy Tools: The most common problem in the orchestration tools is their ability to successfully implement and maintain processes across the organization. Engineers and managers want to choose a tool that addresses the below issues: Dependency management: Lack of dependency management results in maintaining process-to-process and task-to-tasks dependency checks within the codebase along with business logic. This increases development time based on the number of processes or tasks involved in a workflow. Task modification to existing processes: Workflow is defined as each step within the code. Embedding the workflow in the code requires manual effort to fully review the code to make changes to one or more tasks resulting in additional time involved in project enhancements and testing. Troubleshooting: Troubleshooting a particular process should not require the need to understand the dependencies, as it results in additional support time. Restart: It should be fairly easy to restart the failed task or trigger an ad-hoc task. Orchestration should not warrant the need to rerun the entire process if a particular task within the process failed. User-friendly interface to track the progress: Orchestration tools should provide best in class user interface with the details to visualize the tasks involved within the processes in one place. Engineers investigating the issue should have a UI to understand the flow of a particular process. UI should display the current process in execution, the time it's taking, etc. Apache Airflow Airflow is a platform to programmatically author, schedule, and monitor workflows geared at data pipelining. Airflow is used to author workflows as DAGs of tasks. The Airflow scheduler executes tasks on an array of workers while following the specified dependencies. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. Code Development: Authoring a data pipeline using existing tasks involves writing a Python module and deploying it to the Airflow installation. Adding a new kind of task to an airflow installation involves writing and deploying Python code in packages called plugins. This aligns with the primary coding skill set of Data Engineers and scientists, which is Python. Benefits At a high level, process dependency management is easy to develop and maintain via user interface (UI) and support in Airflow, which helps immensely to save time for developers and supporting resources. DAG creation is all in Python scripting language which is the primary coding skillset for Data Engineers. Changes within Airflow can be captured using the GIT repository enabling version controls and change management practices. Airflow has good documentation and an active internal community for answering questions. What are the skillsets required to develop and maintain Airflow? Python is a major skill required to use Airflow. Initial learnings are required to set up Airflow and understand the architecture so it's evident for future maintenance. Can Engineers visualize the Jobs? Yes, Airflow has a Web interface to start and stop processes and also to monitor the current status of the processes. Does Airflow support any integration with Data Quality checkers? DataDog can be integrated with Airflow. How do we make ensure Airflow’s availability? Airflow workers and Web servers can be load balanced. Airflow Scheduler is a single point of failure. How does Airflow scale? Airflow Web server can be scaled based on the number of requests. Airflow workers can be scaled up and down based on requirements.

Apache Airflow And Its Contribution to Enterprise Data Integration

, a powerful open-source search engine, is known for its versatility and flexibility when it comes to handling large volumes of text data. In our continuous effort to improve the user experience and bring advanced search capabilities to a broader audience, we are excited to announce the integration of Manticore Search in , a leading data exploration and visualization platform. Manticore Search Apache Superset This integration is available since Manticore Search version 6.2.0 and allows users to harness the full potential of Manticore Search using the default MySQL connector in Apache Superset, simplifying data analysis and empowering users to create stunning visualizations without the need for extensive technical knowledge. In this article, we’ll provide a step-by-step tutorial that will guide you through connecting Manticore to Apache Superset and adding a chart. Step-by-step tutorial Step 1: Install and Configure Manticore Search Prior to linking with , ensure that Manticore Search has been properly set up and installed on your server. Consult the official Manticore Search guidelines for assistance with . Manticore Search Apache Superset installation and configuration procedures Step 2: Set up Apache Superset To set up Apache Superset, follow the official . Apache Superset installation guide Step 3: Prepare Manticore Search data to visualize In the next step, you need to prepare and create tables in Manticore Search, then load them with data that will help us generate charts and visualize them using Apache Superset. Connect to Manticore using a MySQL client and execute the following queries: Here is the structure we will use: CREATE TABLE product_sales (
id bigint,
product_category text,
sale_date timestamp,
sales_amount float
) Now, let’s add some data to populate the table: INSERT INTO product_sales (product_category, sale_date, sales_amount)
VALUES
('Electronics', 1630454400, 1500),
('Fashion', 1630540800, 1200),
('Home & Garden', 1630627200, 900),
('Electronics', 1630713600, 1800),
('Fashion', 1630800000, 1100),
('Home & Garden', 1630886400, 1000); Step 4: Connecting Manticore Search to Apache Superset Log in to your dashboard and click on “+” in the top navigation bar and choose “Database Connections”. Apache Superset Choose “+ DATABASE” to add a new database. Select “ ” as the database type. MySQL In the configuration page, provide the following details: Database Name: A name for the database (e.g., “Manticore”). Fill up “Host”, “Port”, “Username” and “Display Name” Click on “Connect” to verify the connection and save. Step 5: Creating a Chart and Dashboard In the Apache Superset dashboard, click on “Dashboards” in the top navigation bar and select “New dashboard.” Open the dashboard and click on the “+” icon to add a new chart. Choose the dataset with Manticore Search that connected using the MySQL connector. Select the type of chart you want to create (e.g., bar chart, line chart, pie chart) from the list of available visualization options. For example, we’ll choose a chart. BAR CHART Step 6: Configuring the Chart Use Apache Superset’s query builder or write an SQL query to fetch the data you want to visualize from your Manticore Search database Customize the chart’s appearance, labels, and other settings as needed. Click on “UPDATE CHART” to update the chart. Add name of the chart then click “Save” to permanently save the it in Dashboard. Step 7: Reviewing Your Chart’s Results Here is a sample of the results obtained using the provided data in the newly configured chart. Conclusion The integration of Manticore Search in Apache Superset marks a significant milestone in our ongoing commitment to provide users with advanced search capabilities and seamless data visualization experiences. By following this step-by-step tutorial, you can effortlessly connect Manticore Search to Apache Superset using the default MySQL connector and create engaging, insightful charts that showcase the true value of your data. Don’t miss out on the fantastic opportunities that this powerful combination of Manticore Search and Apache Superset offers. Start exploring your data and unlocking its full potential today! Also published here.

Integrating Manticore Search with Apache Superset 

Recently, we took over a data integration project where the upstream data was delivered to Kafka. Initially, we chose the SpringBoot+Flink approach to process the incoming data (referred to as Solution 1 below). However, after facing efficiency issues in the production environment, we turned our attention to the open-source project SeaTunnel. We discovered that SeaTunnel supports Kafka as a source and started exploring and testing it. In our development environment, we experimented with over 5 million records and achieved an efficiency of around 10,000 records per second. Consequently, we abandoned Solution 1 and adopted SeaTunnel for integrated data processing (referred to as Solution 2 below). During our exploration of SeaTunnel, we identified two methods. Method 2, compared to Method 1, offers a more comprehensive solution that handles various scenarios without concerns about data misalignment caused by unexpected characters within field values. Version Information: SeaTunnel: apache-seatunnel-2.3.2-SNAPSHOT Engines: Flink 1.16.2 Zeta: Consistent with SeaTunnel Comparison Based on Solution 2, two derived methods were developed: After extensive exploration and validation in our production environment, we recommend using Method 2 of Solution 2. Now, let's dive into the main content and explore the two methods in Solution 2, allowing you to experience the magic of SeaTunnel. Solution 1: Kafka Complex JSON Parsing with Spring Boot + Flink There are many online examples available for this approach, so we won't go into much detail here. Solution 2: Kafka Complex JSON Parsing with SeaTunnel Before we proceed, let's take a look at a sample JSON data delivered to Kafka by the upstream party (part of the data has been redacted): "type": "D***",
"headers": null,
"messageSchemaId": null,
"messageSchema": null,
"message": {
    "data": {
        "LSH": "187eb13****l0214723",
        "NSRSBH": "9134****XERG56",
        "QMYC": "01*****135468",
        "QMZ": "1Zr*****UYGy%2F5bOWtrh",
        "QM_SJ": "2023-05-05 16:42:10.000000",
        "YX_BZ": "Y",
        "ZGHQ_BZ": "Y",
        "ZGHQ_SJ": "2023-06-26 16:57:17.000000",
        "SKSSQ": 202304,
        "SWJG_DM": "",
        "SWRY_DM": "00",
        "CZSJ": "2023-05-05 16:42:10.000000",
        "YNSRSBH": "9134****XERG56",
        "SJTBSJ": "2023-06-26 19:29:59.0,00",
        "SJCZBS": "I"
    },
    "beforeData": null,
    "headers": {
        "operation": "INSERT",
        "changeSequence": "12440977",
        "timestamp": "2023-06-26T19:29:59.673000",
        "streamPosition": "00****3.16",
        "transactionId": "000***0006B0002",
        "changeMask": "0FFF***FF",
        "columnMask": "0F***FFFF",
        "transactionEventCounter": 1,
        "transactionLastEvent": false
    }
} Method 1: Parsing without UDF Functions Issue: Field values may contain delimiters, such as ','. This can lead to data misalignment during storage. This method primarily utilizes various transformation plugins available in the official transform-v2. It includes the , , and implementations. Replace Split SQL SeaTunnel Script: (ybjc_qrqm.conf) job.mode = "STREAMING"
job.name = "kafka2mysql_ybjc"
execution.checkpoint.interval = 60000

source {
  Kafka {
    result_table_name = "DZFP_***_QRQM1"
    topic = "DZFP_***_QRQM"
    bootstrap.servers = "centos1:19092,centos2:19092,centos3:19092"
    schema = {
      fields {
        message =  {
            data = {
                LSH = "string",
                NSRSBH =  "string",
                QMYC =  "string",
                QMZ =  "string",
                QM_SJ =  "string",
                YX_BZ =  "string",
                ZGHQ_BZ =  "string",
                ZGHQ_SJ =  "string",
                SKSSQ =  "string",
                SWJG_DM = "string",
                SWRY_DM = "string",
                CZSJ = "string",
                YNSRSBH = "string",
                SJTBSJ = "string",
                SJCZBS = "string"
            }
        }
      }  
    }
	  start_mode = "earliest"
    kafka.config = {
      auto.offset.reset = "earliest"
      enable.auto.commit = "true"
      max.partition.fetch.bytes = "5242880"
      session.timeout.ms = "30000"
      max.poll.records = "100000"
    }
  }
}

transform {
 Replace {
    source_table_name = "DZFP_***_QRQM1"
    result_table_name = "DZFP_***_QRQM2"
    replace_field = "message"
    pattern = "[["
    replacement = ""
  }
  Replace {
    source_table_name = "DZFP_***_QRQM2"
    result_table_name = "DZFP_***_QRQM3"
    replace_field = "message"
    pattern = "]]"
    replacement = ""
  }
  
  Split {
    source_table_name = "DZFP_***_QRQM3"
    result_table_name = "DZFP_***_QRQM4"
    separator = ","
    split_field = "message"
    output_fields = [zwf1,zwf2,zwf3,zwf4,zwf5,nsrsbh,qmyc,qmz,qm_sj,yx_bz,zghq_bz,zghq_sj,skssq,swjg_dm ,swry_dm ,czsj ,ynsrsbh ,sjtbsj ,sjczbs]
  }
  
  sql{
   source_table_name = "DZFP_***_QRQM4"
   query = "select replace(zwf5 ,'fields=[','') as lsh,nsrsbh,trim(qmyc) as qmyc,qmz,qm_sj,yx_bz, zghq_bz,zghq_sj,skssq,swjg_dm ,swry_dm ,czsj ,ynsrsbh ,sjtbsj ,replace(sjczbs,']}]}','') as sjczbs  from DZFP_DZDZ_QRPT_YWRZ_QRQM4 where skssq <> ' null'"
   result_table_name = "DZFP_***_QRQM5"
  }
  
}

sink {
	Console {
		source_table_name = "DZFP_***_QRQM5"
	}
	
    jdbc {
	    source_table_name = "DZFP_***_QRQM5"
        url = "jdbc:mysql://localhost:3306/dbname?serverTimezone=GMT%2b8"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "user"
        password = "pwd"
        batch_size = 200000
        database = "dbname"
        table = "tablename"
        generate_sink_sql = true
        primary_keys = ["nsrsbh","skssq"]
    } 
} Data can be successfully written: Kafka Source Data: Target Data in TiDB: However, when we simulate sending data to Kafka, specifically setting a ',' as part of the 'SJTBSJ' field: Original value: Modified value: 2023-06-26 19:29:59.000 2023-06-26 19:29:59.0,00 Command to produce data to the topic: kafka-console-producer.sh --topic DZFP_***_QRQM --broker-list centos1:19092,centos2:19092,centos3:19092 Data sent: "type": "D***",
"headers": null,
"messageSchemaId": null,
"messageSchema": null,
"message": {
    "data": {
        "LSH": "187eb13****l0214723",
        "NSRSBH": "9134****XERG56",
        "QMYC": "01*****135468",
        "QMZ": "1Zr*****UYGy%2F5bOWtrh",
        "QM_SJ": "2023-05-05 16:42:10.000000",
        "YX_BZ": "Y",
        "ZGHQ_BZ": "Y",
        "ZGHQ_SJ": "2023-06-26 16:57:17.000000",
        "SKSSQ": 202304,
        "SWJG_DM": "",
        "SWRY_DM": "00",
        "CZSJ": "2023-05-05 16:42:10.000000",
        "YNSRSBH": "9134****XERG56",
        "SJTBSJ": "2023-06-26 19:29:59.0,00",
        "SJCZBS": "I"
    },
    "beforeData": null,
    "headers": {
        "operation": "INSERT",
        "changeSequence": "12440977",
        "timestamp": "2023-06-26T19:29:59.673000",
        "streamPosition": "00****3.16",
        "transactionId": "000***0006B0002",
        "changeMask": "0FFF***FF",
        "columnMask": "0F***FFFF",
        "transactionEventCounter": 1,
        "transactionLastEvent": false
    }
} After writing the data, we notice that the data is misaligned: Conclusion: This issue can still occur in production environments when fields contain delimiters like commas in addresses or remarks. Therefore, this approach is not recommended as it poses significant risks to the data. It can be used for handling simple data as an alternative approach. Method 2: Parsing with UDF Functions This method extends SeaTunnel's capabilities using UDF functions ( ), enabling nested JSON source data parsing from Kafka. It simplifies the ST script configuration significantly. https://seatunnel.apache.org/docs/2.3.2/transform-v2/sql-udf ST Script: (ybjc_qrqm_yh.conf) execution.parallelism = 5
job.mode = "STREAMING"
job.name = "kafka2mysql_ybjc_yh"
execution.checkpoint.interval = 60000
}

source {
    Kafka {
        result_table_name = "DZFP_***_QRQM1"
        topic = "DZFP_***_QRQM"
        bootstrap.servers = "centos1:19092,centos2:19092,centos3:19092"
        schema = {
        fields {
             message =  {
                data = "map<string,string>"
             }
        }  
        }
        start_mode = "earliest"
        kafka.config = {
        auto.offset.reset = "earliest"
        enable.auto.commit = "true"
        max.partition.fetch.bytes = "5242880"
        session.timeout.ms = "30000"
        max.poll.records = "100000"
        }
    }
}
transform {
    sql{
        source_table_name = "DZFP_***_QRQM1"
        result_table_name = "DZFP_***_QRQM2"
        query = "select qdmx(message,'lsh') as lsh,qdmx(message,'nsrsbh') as nsrsbh,qdmx(message,'qmyc') as qmyc,qdmx(message,'qmz') as qmz,qdmx(message,'qm_sj') as qm_sj,qdmx(message,'yx_bz') as yx_bz,qdmx(message,'zghq_bz') as zghq_bz,qdmx(message,'zghq_sj') as zghq_sj,qdmx(message,'skssq') as skssq,qdmx(message,'swjg_dm') as swjg_dm,qdmx(message,'swry_dm') as swry_dm,qdmx(message,'czsj') as czsj,qdmx(message,'ynsrsbh') as ynsrsbh, qdmx(message,'sjtbsj') as sjtbsj,qdmx(message,'sjczbs') as sjczbs  from  DZFP_DZDZ_QRPT_YWRZ_QRQM1"
    }
}

    
sink {
    Console {
        source_table_name = "DZFP_***_QRQM2"
    }
    jdbc {
        source_table_name = "DZFP_***_QRQM2"
        url = "jdbc:mysql://localhost:3306/dbname?serverTimezone=GMT%2b8"
        driver = "com.mysql.cj.jdbc.Driver"
        user = "user"
        password = "pwd"
        batch_size = 200000
        database = "dbname"
        table = "tablename"
        generate_sink_sql = true
        primary_keys = ["nsrsbh","skssq"]
    } 
} Executing the script and checking the results, we can see that the data is not misaligned and remains in the original field ('sjtbsj'): The specific UDF function is implemented as follows. Maven dependencies are as follows: <dependency>
    <groupId>org.apache.seatunnel</groupId>
    <artifactId>seatunnel-transforms-v2</artifactId>
    <version>2.3.2</version>
    <scope>provided</scope>
  </dependency>

  <dependency>
    <groupId>org.apache.seatunnel</groupId>
    <artifactId>seatunnel-api</artifactId>
    <version>2.3.2</version>
  </dependency>

  <dependency>
    <groupId>cn.hutool</groupId>
    <artifactId>hutool-all</artifactId>
    <version>5.8.20</version>
  </dependency>

  <dependency>
    <groupId>com.google.auto.service</groupId>
    <artifactId>auto-service-annotations</artifactId>
    <version>1.1.1</version>
    <optional>true</optional>
    <scope>compile</scope>
  </dependency>

  <dependency>
    <groupId>com.google.auto.service</groupId>
    <artifactId>auto-service</artifactId>
    <version>1.1.1</version>
    <optional>true</optional>
    <scope>compile</scope>
  </dependency>
</dependencies> UDF-specific implementation in Java: import cn.hutool.core.util.StrUtil;
import cn.hutool.json.JSONObject;
import cn.hutool.json.JSONUtil;
import com.google.auto.service.AutoService;
import org.apache.seatunnel.api.table.type.BasicType;
import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
import org.apache.seatunnel.transform.sql.zeta.ZetaUDF;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

@AutoService(ZetaUDF.class)
public class QdmxUDF implements ZetaUDF {

    @Override
    public String functionName() {
        return "QDMX";
    }

    @Override
    public SeaTunnelDataType<?> resultType(List<SeaTunnelDataType<?>> list) {
        return BasicType.STRING_TYPE;
    }

    // list Example of parameters: (which is the data parsed by Kafka)
    //SeaTunnelRow{tableId=, kind=+I, fields=[{key1=value1,key2=value2,.....}]}
    @Override
    public Object evaluate(List<Object> list) {
        String str = list.get(0).toString();
        //1 Remove the prefix
        str = StrUtil.replace(str, "SeaTunnelRow{tableId=, kind=+I, fields=[{", "");
        //2 Remove the suffix
        str = StrUtil.sub(str, -3, 0);
        // 3 build Map key value
        Map<String, String> map = parseToMap(str);
        if ("null".equals(map.get(list.get(1).toString())))
            return "";
        // 4 return the value of the key
        return map.get(list.get(1).toString());
    }

    public static Map<String, String> parseToMap(String input) {
        Map<String, String> map = new HashMap<>();
        // Remove the curly brackets and remove them during the string stage.
        // input = input.replaceAll("[{}]", "");
        // Split key-value pairs
        String[] pairs = input.split(", ");

        for (String pair : pairs) {
            String[] keyValue = pair.split("=");
            if (keyValue.length == 2) {
                String key = keyValue[0].trim().toLowerCase();
                String value = keyValue[1].trim();
                map.put(key, value);
            }
        }
        return map;
    }
} Then, for packaging, use the following command: mvn -T 8 clean install -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true Check the directory to see if the @AutoService annotation generates the corresponding SPI interface. META-INF/services If it is as shown below, then the packaging is successful! If not, the packaging fails, and the UDF function cannot be used. You can refer to my packaging plugin: <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-site-plugin</artifactId>
      <version>3.7</version>
      <dependencies>
        <dependency>
          <groupId>org.apache.maven.doxia</groupId>
          <artifactId>doxia-site-renderer</artifactId>
          <version>1.8</version>
        </dependency>
      </dependencies>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.8.1</version>
      <configuration>
        <annotationProcessorPaths>
          <path>
            <groupId>com.google.auto.service</groupId>
            <artifactId>auto-service</artifactId>
            <version>1.1.1</version>
          </path>
        </annotationProcessorPaths>
      </configuration>
    </plugin>
  </plugins>
</build> Finally, place the packaged JAR file in the $ directory. Since my UDF function depends on a third-party JAR, it should be uploaded together. If using the Zeta cluster, you need to restart the Zeta cluster for it to take effect. For other engines, it will take effect in real time. {SEATUNNEL_HOME}/lib The successful upload will look like this: Note: The "hutool-all" JAR file can be included in the project. I uploaded two for convenience. java_study In conclusion, it is recommended to to parse the nested JSON data from Kafka sources. use the extension of UDF functions Also published . here

Resolving Complex JSON in Kafka Source Using Apache SeaTunnel

Apache Pulsar has always seen itself as cloud native. It’s right there on the home page: “Apache® Pulsar™ is an open-source, distributed messaging and streaming platform built for the cloud” Pulsar certainly embodies many cloud-native principles, including the separation of compute from storage where the Pulsar broker handles the serving of messages and Apache BookKeeper handles the storage of them, and the idea of stateless components that can be scaled up and down at will like the Pulsar broker. However, it has never really delivered on the ultimate promise of the cloud: autoscaling. At least, not until today. I am excited to introduce Kubernetes Autoscaling for Apache Pulsar, or KAAP. KAAP is a Kubernetes operator that is available on It includes all the niceties that you would expect from a Kubernetes operator. Once you have the operator installed in your Kubernetes cluster, you can get a fully functional Pulsar cluster by applying a single custom resource definition (CRD) like this: OperatorHub. apiVersion: kaap.oss.datastax.com/v1alpha1
kind: PulsarCluster
metadata:
  name: pulsar
spec:
  global:
    name: pulsar
    image: 'apachepulsar/pulsar:3.0.0' Pulsar has multiple components: ZooKeeper, BookKeeper, brokers, and proxies. The operator handles configuring all of them with the CRD. Because the operator works at the cluster level, you don’t have to worry about how the components work together. The operator takes care of that for you. PulsarCluster Automatic staged upgrades Having operated a production cloud service for Pulsar longer than anyone I know of (first at and now with DataStax Astra Streaming), I know firsthand how long it takes to do a proper upgrade of a Pulsar cluster. While you could just YOLO upgrade all the components at once, the recommended and safest method is to upgrade each component one at a time, ensuring each component has successfully upgraded before moving on to the next. Kesque This is the process that we follow when upgrading the Pulsar clusters in our Astra Streaming service because the availability of our customer service is paramount, but it is very time-consuming for our production engineering team. In fact, this is the number one request they have: can you make the upgrades smarter and more automatic? Well, with KAAP, we have done just that. With a single line change in the CRD, you can trigger a staged upgrade of your Pulsar cluster. The KAAP operator will orchestrate a careful, staged upgrade of your cluster. Of course, you should monitor the metrics during the upgrade, but you can let the KAAP operator drive (no sleeping at the wheel). PulsarCluster Open source autoscaling Automatic staged upgrades are nice, but that’s not what I am most excited about with KAAP. The concept of elastic resources is a key promise of the cloud. It is why many of the AWS services have “elastic” in their name. I (actually ) did a quick search and there are at least 10 of them. ChatGPT The cloud enables you to acquire and release resources on demand, so you can use autoscaling techniques to match the number of resources your application is using to the load on it. As the load increases, you can automatically add more resources. As the load decreases, you can free up resources. Since cloud providers charge for the usage of resources, autoscaling can help you make the most efficient use of your cloud spend. I am sure we can all think of examples of cloud services that feature autoscaling behind the scenes. Most of those “elastic” AWS services work like that. However, the implementation of the autoscaling is proprietary, locked in a vendor’s private repo. Sometimes they will talk about how they have implemented autoscaling, like in this about AWS DynamoDB, or this about how Confluent Cloud makes Kafka elastic. But no one, to my knowledge, has made their autoscaling implementation open source, freely available to all users under a truly open source license. That’s exactly what we are doing with KAAP. paper blog post KAAP adds sophisticated autoscaling capabilities to your Pulsar (or , our Pulsar distribution) under an Apache 2.0 license. You can check it out in our . If you like it, remember to give it a star. Luna Streaming GitHub repository Pulsar + auto scaling: A perfect match KAAP gives you a way to add autoscaling to Pulsar when running in Kubernetes. Not some simple, Horizontal Pod Autoscaler (HPA) tacked on to a Pulsar broker deployment, but a sophisticated auto scaler that works with the existing cloud-native components of Pulsar to provide a truly elastic streaming and messaging system. In my opinion, this is a combination that can’t be beat. Stateless broker scaling Let’s take a look at the two main autoscaling dimensions of KAAP. First, we’ll look at Pulsar brokers. As you might know, Pulsar brokers are stateless, which means you can scale them up and down easily, as required. But what you might not know is that Pulsar has a built-in broker load-balancing mechanism that continuously monitors CPU, memory, and network bandwidth on each broker. Using that information and one of the configurable load-balancing algorithms, Pulsar will move topics from broker to broker to prevent a broker from being overloaded. A naive autoscaling solution would be to configure a Kubernetes Horizontal Pod Autoscaler (HPA) on the brokers and when some metric like CPU gets high, scale up another broker pod. But this might not actually be necessary because the Pulsar broker load balancer could decide to shift topics to even out the load at the same time. Now you’ve scaled up a broker pod that isn’t needed because the Pulsar load balancer has balanced things out. So now the HPA decides to scale down the new broker pod, which causes any new topics that had been created on it to get moved to an existing broker.  As you can imagine, the Pulsar load balancer and an HPA can create a thrashing mess of brokers going up and down and topics being shifted from broker to broker. KAAP avoids this problem by integrating directly with the Pulsar load balancer. KAAP scales up the brokers when the cluster-wide metrics from the Pulsar load balancer suggest that the cluster is nearing capacity, not when a single broker pod is busy. And it only scales down a broker if the usage of the whole cluster falls below a configured threshold. The KAAP operator works with the Pulsar broker load balancer, not against it. Scaling storage up and down: Remarkable! Scaling the compute (or serving) layer of Pulsar is great, but it’s not enough for a true autoscaling implementation. Sure, the number of messages (or events) that need to be processed can vary over time, but what about the number of messages that need to be stored? We’ve probably all had to deal with a backlog of messages building up because of a failure of a downstream system. As the outage drags on, the available storage on the streaming system starts to run low. This is a scenario where Pulsar and its reliance on BookKeeper shines. To add storage capacity to a Pulsar cluster, you just need to add a new BookKeeper node, affectionately called a “bookie.” Because BookKeeper storage is based on segments of topics and not whole topics, the new bookie is immediately available to relieve the pressure on storage, with any painfully rebalancing of topics or any other operational intervention. KAAP can, of course, handle this case for you. It constantly monitors the disk usage of the bookie nodes, and if available storage is getting low, it will scale up a new bookie node. This is not all that remarkable (at least for Pulsar). It’s pretty straightforward to add a new replica in Kubernetes, even if it is stateful and backed by persistent volumes. But how about when the outage is over and the backlog has cleared? Are you now stuck with an extra bookie node consuming resources that aren’t really needed anymore? Not with KAAP, you’re not. Once the BookKeeper storage drops below a configured threshold, the KAAP operator will carefully orchestrate the removal of the unneeded bookie node. It does this in a very safe way, ensuring that no messages are lost and that the required replication factor is maintained at all times. For example, if you have configured Pulsar to keep three copies of each message (write and ack quorum both equal three), KAAP interacts with the BookKeeper to copy messages from the bookie that is being scaled down to other bookies in order to ensure there are at least three copies of the message available. Once that has been completed successfully, it will proceed with removing the unneeded bookie. With KAAP you get automatic scaling of the storage in your Pulsar cluster both up and down so that you can optimize the storage usage in your cluster and not get stuck with idle capacity after an unfortunate production incident. I don’t know about you, but I think that’s pretty remarkable. Zone awareness and migration tools KAAP can do staged upgrades and a smart scaling of your Pulsar cluster. But there’s more. To operate a highly available cluster in a cloud provider, it’s important to take availability zones (AZ) into consideration. If you don’t spread your components, especially BookKeeper, across availability zones, you won’t be able to survive an AZ failure and provide multiple nines of availability. Luckily Pulsar has great built-in capabilities like rack awareness to support high-availability deployments. The tricky part is that to set it properly you need to both configure Kubernetes correctly with zone awareness and also configure Pulsar. The KAAP operator has you covered by introducing the concept of resource sets, which enable you to group components and give them rack awareness. The KAAP operator will automatically apply both the Kubernetes and Pulsar configurations based on your declarative configuration of resource sets. Resource sets are flexible, allowing you to support a variety of Pulsar deployment options. And what if you are already running your Pulsar using a Helm chart or maybe just some Kubernetes manifest magic? KAAP has a migration tool to help you out. You can point the migration tool at your existing Kubernetes Pulsar deployment and it will automatically generate a matching CRD configuration that you can use to have the KAAP operator take over operating your cluster for you. The KAAP Stack The KAAP operator has lots of great features, turbocharging your regular Pulsar cluster into a well-oiled, highly available, autoscaling machine. But as someone who has operated production Pulsar clusters for a long time, I know that there are lots of other considerations for creating a production Pulsar cluster, such as TLS certificate management, authentication, and monitoring. That’s why we have included what we call the KAAP stack with the operator. It’s an umbrella Helm chart that installs the KAAP operator along with critical production tools, including: Cert Manager Keycloak Prometheus Stack (Grafana) Pulsar Grafana dashboards These are must-have tools when running our production Pulsar clusters and we didn’t want to leave you high and dry, so we pulled them all together and integrated them into one convenient package. Why use KAAP? So you’ve heard about all the great features of Kubernetes Autoscaling for Apache Pulsar. You can create an entire Pulsar cluster with a single CRD. You can put your upgrades on autopilot and let the KAAP operator perform safe, staged upgrades. You can automatically scale Pulsar brokers up and down based on the Pulsar broker load balancer saying the brokers are getting overloaded, and you can scale BookKeeper nodes up and down (!) safely based on the storage requirements of your clusters. You can easily configure your cluster for availability zone awareness for high availability. And it even includes a migration tool so you can easily move from your old Helm-based deployment to a turbo-charged, KAAP operator-based one. So—lots of great features, but what are the benefits of KAAP? I can think of several: Easily configure and operate highly available Pulsar clusters running in Kubernetes leading to less effort for you and your production teams Dramatically simplify scaling Pulsar to match changing demands by autoscaling the cluster resources to match the demand Reduce the total cost of ownership of a Pulsar cluster by eliminating provisioning to peak loads or over provisioning as result of production incidents Avoid vendor lock-in by using all open source technologies In my opinion, releasing KAAP is a truly innovative moment in the streaming and messaging space. No other open-source project combines the streaming and messaging power of Apache Pulsar with the ultimate promise of cloud computing: fully elastic, autoscaling. I’m looking forward to you trying it out. Hop into GitHub Discussions in our repo and let us know what you think! Find out more If you want to take a technical deep dive into KAAP, take a look at . You can find the complete documentation for KAAP . And is the GitHub repository. this blog post here here By Chris Bartholomew, DataStax This story was distributed by Datastax under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author

Fulfilling the Cloud Native Promise: Autoscaling for Apache Pulsar Is Here!

Streaming que cambia el juego: Apache Pulsar presenta el escalado automático

Streaming qui change la donne : Apache Pulsar présente l'autoscaling

गेम-चेंजिंग स्ट्रीमिंग: अपाचे पल्सर ने ऑटोस्केलिंग की शुरुआत की

革新的なストリーミング: Apache Pulsar が自動スケーリングを導入

Streaming revolucionário: Apache Pulsar apresenta escalonamento automático

Truyền trực tuyến thay đổi trò chơi: Apache Pulsar giới thiệu Autoscaling

改变游戏规则的流媒体：Apache Pulsar 推出自动缩放功能

Schemas are an essential part of any They’re metadata that define the shape of the data and the properties’ names and data types. Schemas are helpful in two ways. First, they provide a fixed blueprint for the data format, which can prevent badly formed data from being used within the context of the schema. Second, schemas let users understand how to parse the data and what to expect from its properties. data platform. is an open source, distributed publish-subscribe (pub-sub) messaging system that enables server-to-server data transportation . Messages are sent via topics, with the producer putting messages onto these topics to be read by the consumer. A user can define a schema for a topic stored in Pulsar’s schema registry. When messages are added to the topic, the Pulsar broker checks that the message conforms to the schema and ensures only valid messages are sent. The schema acts as a contract between the producer and consumers, enabling both parties to know the exact format of the data. Apache Pulsar using schema Over time, as applications evolve, it may be that the data produced by the application changes too. However, schema changes can affect the downstream consumers of the data that expect data in a specific format. Without a way to manage schemas between producers and consumers, it’s difficult to make changes to the data that is written in the messages or events without risking breaking downstream applications. To avoid this kind of issue, the schema of the messages must also evolve as new properties are added and allow consumers to understand data in both the old and new formats. This concept is known as schema evolution and Pulsar supports it. This article discusses why schemas evolve before diving into how Pulsar implements and supports schema evolution. Why Schemas Evolve Schemas provide context about raw data. They often describe a particular entity in a system, encompassing all the properties of that entity. For example, you may have an application that signs up users. It stores user details such as names, email addresses and ages. There would be a user schema that describes the underlying data and provides context, such as the name of the field and the type of data in it. The schema may look as follows: Now, say you wanted to expand the data captured to include address data and support direct mailing to the users. You would then need to expand the schema to include the new fields for capturing addresses, such as the first line of the address, city and ZIP code. After including these new fields, the schema will look like this: This is a simple form of schema evolution, as the original fields haven’t changed, and only new fields have been added. In most cases, this shouldn’t be a breaking change for downstream consumers, as consumers can continue as if the new fields were absent. The consumers would just need to be updated to consume and use the new properties. However, sometimes existing fields need to be amended to support new functionality. For example, say that users were uncomfortable giving an exact age, and instead, you change the application to capture age ranges like 18-24, 25-39,40-49 and 60+. The age column would need its data type amended from integer to string. This is a more complex schema evolution, as this could break downstream consumers who are processing the age property and expecting it to be a number or parsing the number using a strictly typed language like Java. They could also perform numeric calculations on the property, which would no longer work in its new format. To overcome this challenge, data platforms can support schema evolution to handle scenarios like this. Pulsar recognizes the importance of schema to data processing; in fact, it treats it like a first-class citizen by including built-in schema evolution support. Let’s look at how Pulsar does this. How Pulsar Supports Schema Evolution Pulsar defines schemas in a data structure called `SchemaInfo`. This is a Pulsar-specific data structure, which is itself a schema, that describes the schema of messages transported via Pulsar. Each `SchemaInfo` belongs to a topic (message channel) and describes the messages to be produced and consumed using that topic. Each `SchemaInfo` has a type that details the type of schema being used. This can be anything from an integer, a string, or a complex schema such as Avro or Protobuf. To support , Pulsar uses schema compatibility checks to compare an incoming schema to a topic’s known schema(s). Schema compatibility checks occur when a producer or consumer attempts to connect to a particular topic. The strategy is chosen while configuring the broker, with the value `schemaCompatibilityStrategy`. The goal is to check the `SchemaInfo` provided by the connecting client with existing `SchemaInfo` to see if they are compatible. schema evolution Pulsar supports eight different types of , which you can set based on the requirements of the change. These strategies also come with rules outlining what changes can be made and pointers around which clients to update first. (Check out the documentation linked above for a deeper explanation of each compatibility strategy). schema compatibility strategies Returning to the earlier examples, let’s implement the schema changes using Pulsar’s compatibility strategy. First, start with the initial user schema (without the address). This will be V1 of your schema. So, when you implement a Pulsar producer or consumer for the first time, the `SchemaInfo` for this version will be stored, and the producer and consumer will work as expected. Next, you want to add the new address fields to your user schema. The first step is to consult the schema compatibility strategies and determine which one is best for this change. Using the “changes allowed” column in the documentation, you’re looking for any strategy that allows the addition of new fields. This gives you BACKWARD, BACKWARD_TRANSITIVE, FORWARD, and FORWARD_TRANSITIVE. BACKWARD should be used when there’s no guarantee that consumers using an older version can understand the new schema. FORWARD is used when the consumers on the latest schema version might not be able to read data within older versions. If you want to update all of the consumers first to use the new schema, use a BACKWARD strategy. Otherwise, FORWARD is best. Looking at the bigger picture, Pulsar refers to the entire act of evolving a topic’s schema as . It’s the combination of a producer or consumer providing `SchemaInfo`, the chosen compatibility strategy for the topic, and the broker deciding what to do. schema verification Conclusion It’s rare for schemas to stay the same forever. As new features are introduced into applications, schemas often have to evolve to support these features. However, keeping the data's producers and consumers in sync can often be a challenge when schemas are modified. Pulsar’s built-in schema evolution concepts help deal with these changes. Using schema compatibility strategies, it can define the rules of how different compatible versions of a schema are. Pulsar uses this in conjunction with a schema verification process that then uses these rules to determine which schemas can be used by a consumer when connecting to a particular topic. Also published here.

How to Manage Schema Change with Apache Pulsar

Cómo administrar el cambio de esquema con Apache Pulsar

Comment gérer le changement de schéma avec Apache Pulsar

अपाचे पल्सर के साथ स्कीमा चेंज को कैसे प्रबंधित करें

Apache Pulsar を使用してスキーマ変更を管理する方法

Como gerenciar a mudança de esquema com o Apache Pulsar

Cách quản lý thay đổi lược đồ với Apache Pulsar

如何使用 Apache Pulsar 管理模式更改

Apache Cassandra is becoming the best database for handling JSON documents. If you’re a Cassandra developer who finds that statement provocative, read on. In a previous article, to mold Cassandra into a developer experience more idiomatic to the way developers think, thus improving developer productivity while preserving reasonable database performance and scale. It’s a great hypothesis and one that needs to be tested in the context of a particular developer idiom and developer community. we discussed using data APIs and data modeling , an object data mapping library usually used with the MongoDB driver, is an open source project with a significant community of JavaScript developers around it. At the , the open source API data gateway designed to work with Cassandra, we’ve partnered with Mongoose, and we’re working on an upcoming that will be released together with a that will work through that JSON API to connect to Cassandra. This creates an end-to-end stack for Mongoose developers that is fully open-source. It’s game-changing for Mongoose developers, and opens an important new chapter for Cassandra. Mongoose Stargate project JSON API version of Mongoose In this article, I’ll discuss how to provide a developer-friendly JSON idiom using Cassandra together with Stargate, and how we’re working to do just that for Mongoose developers. The Goldilocks of JS Communities In October 2022, we released a new version of Stargate. With the new , individual APIs are no longer embedded in the core Stargate coordinator code, but instead separated out into individual services. This improves Stargate’s operational efficiency; individual API services can now be deployed and scaled independently. This also makes new API services easier to develop. As long as they abide by the service boundary, these services can be developed in parallel with and independent of core Stargate development work. Version 2 We then looked for a truly idiomatic experience we could deliver to developers. With 18 million developers, JavaScript is the world’s most popular programming language, and JSON is the standard way that JavaScript developers structure data. However, 18 million people is not a community; it is many communities. We needed the “Goldilocks” of JavaScript communities — big enough to be significant, but small enough to be focused. We found the right community built around Mongoose, an object data mapper library used with applications that connect to MongoDB. Mongoose has several important characteristics: It’s JavaScript-centric It enjoys broad adoption, with roughly 2 million GitHub repositories listing Mongoose as a dependency Mongoose creator ’s active leadership provides a clear focus Valeri Karpov It’s an open source project that has lacked an open source database since MongoDB’s decision to move to a shared source model with its Server Side Public License Developers don’t really interact directly with a database so much as a data model. In Stargate’s original Document API, the API handles JSON by making it look like a traditional Cassandra table. This puts a burden on JSON-oriented developers to think in terms of Cassandra data structures, and puts a burden on Cassandra’s row-oriented indexing logic because a JSON document gets spread across multiple rows. Our new JSON API departs from this data model, and instead relies on a data model we call “super shredding.” You can learn more about super shredding by watching Aaron Morton’s talk at the recent event. In short, we take advantage of Cassandra’s wide-column nature to store one document per row, knowing that a Cassandra row can handle even very large documents. We also have a set of columns in that row that are explicitly for storing standard metadata characteristics of a JSON document. Now we have something more easily indexable, as well as a means of preserving and retrieving metadata. Cassandra Forward We will then front this data model with our new JSON API, using the same specification that Mongoose uses as our guiding requirement for which calls the API needs to support. When complete, this should enable any of the more than 2 million Mongoose-dependent applications to run against open-source Cassandra or DataStax’s hosted Cassandra service, , with just a configuration change. mQuery Astra DB With Mongoose and the new JSON API, we’ll provide a fully idiomatic experience to JSON-oriented JavaScript developers, giving them the underpinning an authentic JSON data model. scale and performance of Cassandra Mongoose creator Karpov also spoke at the recent Cassandra Forward event (you can watch the replay ), demonstrating a simple e-commerce application that uses the Stargate version of Mongoose, open source Stargate and the DataStax Enterprise (DSE) version of Cassandra. You can download the working code for this application and the supporting platform pieces from . While we have enough code to run this application, we are not yet code complete. For example, we run against DSE right now because we need (SAI), which works with DSE and is planned for release in Cassandra 5.0 later this year. here GitHub storage-attached indexing Contributing Back to Cassandra Cassandra isn’t a static piece of software; it’s a vibrant and evolving open-source project. So we are also continuing a longstanding Cassandra tradition of using that emerge client-side to foster changes on the database side.o  Similarly, Stargate’s Mongoose work has prompted a set of proposals for Cassandra around global sort and advanced query filtering that will not only make Stargate’s JSON API and Mongoose client better, but will add powerful new features to Cassandra Query Language. This is a great reminder that data engineers and application developers are not two different communities, but complementary cohorts of the extended Cassandra community. features like SAI And JSON is just the first step. Essentially, what we will have done is to take the building blocks of Cassandra, Stargate and a reasonably efficient Cassandra data model and build a document database that you interact with through a JSON API. In other words, we’ve used super shredding to create a purpose-built database that better serves the community of Mongoose developers. With the modular architecture of , and the proof point of Mongoose for the idiomatic approach, we are ready to take on new developer communities that organize around a particular software development idiom. The process by which we’ve harnessed Cassandra for Mongoose is repeatable — and it’s one that we will repeat. In so doing, we dramatically expand the number of developers and use cases that Cassandra can address, which is the sort of goal worthy of an open-source project. Stargate v2 . Learn more about DataStax By Mark Stone. Mark is product manager at DataStax. He's a technology veteran with many years of experience in product management, program management and people management. Always working as part of the connective tissue between business stakeholders and technical stakeholders, Mark loves championing the developer experience in technology platforms and helping organizations meet developers where they are.

How Mongoose Will Bring JSON-Oriented Developers to Apache Cassandra

Cómo Mongoose traerá desarrolladores orientados a JSON a Apache Cassandra

Comment Mongoose amènera les développeurs orientés JSON vers Apache Cassandra

Mongoose JSON-उन्मुख डेवलपर्स को Apache कैसेंड्रा में कैसे लाएगा

Mongoose が JSON 指向の開発者を Apache Cassandra にもたらす方法

Como a Mongoose trará desenvolvedores orientados a JSON para o Apache Cassandra

Mongoose sẽ đưa các nhà phát triển định hướng JSON đến với Apache Cassandra như thế nào

Mongoose 如何将面向 JSON 的开发人员带到 Apache Cassandra

This article explains an alternate option for Apache Airflow to manage or run data quality checks. is an open-source platform that has become increasingly popular for managing data pipelines. Airflow And why not, it provides a convenient way to automate and organize the workflow of tasks related to data processing, making it easier for organizations to manage their data pipelines efficiently. However, while Airflow has proven to be a powerful tool for managing data pipelines, it has significant limitations regarding managing data quality. Needless to say, compromised data quality is of no good to the organization. In this blog, we’ll discuss some challenges while using Airflow and why you might need to reconsider this choice for data quality-related tasks. By the end of the blog, we hope you’ll determine if Airflow is the right fit for your data needs. Data quality is an essential aspect of any data pipeline, and it's critical to ensure that data is accurate, consistent, and free from errors. The data quality must be monitored and maintained throughout the entire data pipeline, from the source to the final destination. Interestingly, Airflow does not have built-in features for managing data quality, so organizations must use additional tools and techniques to maintain the quality of their data. Let us discuss some of Airflow's limitations when managing data quality and why it's not the best choice for organizations that need to ensure the accuracy and consistency of their data. We’ll also discuss alternative tools and techniques organizations can use to manage data quality in their pipelines. By understanding the limitations of Airflow and the importance of data quality management, organizations can make informed decisions about how they manage their data pipelines and ensure that they use the right tools for the job. So, let us see why Airflow might not be the best choice for your data quality needs. Reasons Not to Use Airflow for Data Quality: 1. Lack of Built-in Data Quality Control Features: Airflow was designed to automate and manage data pipelines, but it does not have built-in features for managing data quality. Without these features, it becomes difficult to detect errors and inconsistencies in the data, leading to incorrect insights and decision-making based on inaccurate data. This would mean that organizations must use additional tools to validate their data and ensure that it's accurate and consistent. 2. Inefficient Data Validation Processes: Data validation is a crucial step in data quality management. This leads to a higher risk of errors and inconsistencies in the data, which can compromise the accuracy and consistency of the data. The inefficient data validation processes in Airflow can also lead to a longer time to complete data processing tasks, impacting the organization's ability to make timely and informed decisions. 3. High Risk of Data Errors and Inconsistencies: The lack of built-in data quality control features and the inefficient data validation processes in Airflow increases the risk of data errors and inconsistencies. These errors and inconsistencies can compromise the accuracy and consistency of the data, leading to incorrect insights and decision-making based on inaccurate data. This can have serious consequences for the organization, including lost revenue, damaged reputation, and decreased customer trust. 4. Data lineage is an important aspect of data quality management as it provides a clear understanding of the origin, transformation, and flow of data throughout a pipeline. With Data lineage, data engineers derive the what, where, and how of the data. What data is being processed, where it comes from, and how it's transformed? In Airflow, the lack of built-in lineage information can make it difficult for data engineers to fully understand the impact of any changes to data quality checks or the data pipeline as a whole. This can make it difficult to trace the root cause of data quality issues. Furthermore, without lineage information, data engineers may struggle to identify dependencies between different data quality checks and the impact of changing those checks. This can lead to errors in the data pipeline and negatively impact data quality. Now that we understand its incapability in data lineage and the overall data quality, let us see the major challenges of using Airflow in a distributed system. Challenges in Managing Data Quality in a Distributed System: Data quality management in distributed systems presents unique challenges that require specialized features. Some of the challenges include the following: Lack of coordination across distributed teams: Airflow does not have the built-in capabilities to manage the coordination of data quality checks across multiple teams. Imagine a software company with a distributed data team responsible for different data sources, such as user data, product data, and usage data. Each department has its own data quality checks and procedures in place, but there is no centralized tool or process to coordinate data quality management across the teams. This results in data quality issues slipping through the cracks and inconsistent data quality across the different data sources. Real-time data quality management: Airflow may not have the ability to keep up with the speed and scale of data processing in these systems, leading to gaps in data quality management. A SaaS company that processes large amounts of user data in real time would need robust speed and scale for data processing. Airflow is not capable of keeping up with this. As a result, data quality issues are not identified and corrected in a timely manner, leading to gaps in data quality management. Inconsistency managing multiple data sources: Airflow may not have the necessary tools to manage data quality consistency across multiple data sources in a distributed system. A software company has multiple data sources, including user data from its SaaS platform, product data from its APIs, and usage data from its analytics tools. Each data source has different data quality standards, making it difficult to maintain consistent data quality across all sources. Data quality Issues while handling distributed systems: Airflow may not have the built-in tools to efficiently address data quality issues in a timely manner. When data quality issues are identified, fixing them can be difficult and time-consuming, especially in a system with multiple teams and data sources. Airflow does not have the built-in tools to efficiently address data quality issues in a timely manner, making it a challenge to maintain high-quality data in the system. Airflow is not designed for data quality. Let's see which alternatives can assist you if you are looking for a solution to all these challenges. Better Alternatives to Ensure Data Quality: There are several alternative tools and techniques that organizations can use to manage data quality in their data pipelines. These tools are specifically designed for data quality management and provide data validation, profiling, and cleansing features. It's all about choosing the one that suits best your needs. Here we suggest the most popular alternatives: Talend provides a user-friendly interface with drag-and-drop functionality, making it easy for data engineers to create and execute data quality checks. It also offers a range of data quality tools to ensure accurate and consistent data and can be integrated with multiple data sources, unlike Airflow. Microsoft SQL Server Integration Services (SSIS), with scalability and performance to handle large volumes of data, offers built-in data quality tools for cleaning, standardizing, and validating data. Its capability to seamlessly integrate with top technologies such as Azure, and Power BI makes it the go-to choice for organizations seeking to maximize their data capabilities while minimizing integration challenges. Our favorite: dbt-test! It's a testing framework specifically designed for data engineers and data scientists. It's simple to use, with a minimal learning curve, and provides a range of tests to validate data quality and accuracy. Unlike Airflow, dbt-test is designed specifically for data quality management, making it a more efficient and effective solution. With its ability to handle complex data quality scenarios, dbt-test is the preferred choice for many organizations looking to manage data quality in their data pipelines. Our favorite is the combination of Airflow and dbt-test. Why you may ask. Airflow and dbt together can enable each member of the data team to focus on their specific areas of expertise and have a clear understanding of who should be involved in any troubleshooting when issues arise. How does that happen? Airflow and dbt each serve a specific purpose in the data engineering workflow, with Airflow being responsible for orchestrating jobs related to data extraction, loading, and machine learning processes, while dbt focuses on data transformation using SQL. When used together, these two tools can complement each other to provide the best results for data teams. ‍ So, always evaluate the alternatives and make a wise decision in choosing the right tool since it's going to make or break the data quality. Adapting the Right Tool Resolves Data Quality Issues: While Airflow has some features that make it useful for data engineering, it's not the best choice for data quality management. The lack of built-in tools and the difficulty in coordinating and managing data quality across distributed teams make it an inefficient solution for data quality management. Alternatives provide more comprehensive data quality management capabilities, making them a better choice for data quality management in distributed systems. Airflow with dbt-test, in particular, stands out for its simplicity and specificity to data quality management, making it an efficient and effective solution for organizations since this combination enables data teams to work collaboratively and deliver reliable data to end-users, ensuring high-quality data in their pipelines. What steps are you taking to ensure bullet-proof data quality in your distributed systems? Also published here

Apache Airflow: Is It a Good Tool for Data Quality Checks?

An extraordinary computer science breakthrough called Accord is bringing globally available, general-purpose ACID transactions to the next Cassandra release Let’s get the best part out first. are coming to Apache Cassandra. ACID transactions Globally available, general-purpose transactions that work the way Cassandra works. This isn’t some trick with fine print or the application of old techniques. It’s due to an extraordinary computer science breakthrough called from a team at Apple and the University of Michigan. It will help Cassandra change how we think of data by opening new use cases. Accord Here’s what this means for those who aren’t steeped in the ins and outs of the Cassandra project and its features. Nothing is more important than how fast you can put an application into production. But developers that want to put Cassandra’s scale, resilience, and legendary multi-data center support to work on something like financial transactions had to code a bunch of complicated workarounds into their apps. The trade-offs versus using, say, Oracle were significant. With Accord? No trade-offs. Cassandra will now that has made it amazing while shifting transaction support to the database, significantly reducing code complexity. support everything The Eye of the Observer A database system has essential functions, such as and being available for query. Managing changes in data isn’t always a database function. In the case of many NoSQL systems, the burden of change management is deferred to the user. The observer of the data change is who assigns the importance of exclusivity. storing data reliably Suppose the point is to accumulate data as given. In that case, the observer must know the data was received and stored durably — for example, stock ticker data where every data point is unique and cumulative. There is no need for exclusivity. In more sensitive operations, the observer of a data change needs to feel like they are the only process using the database. It’s a concept in computer science called “isolation”; it’s the “I” in ACID  (atomicity, consistency, isolation, durability). A classic example is a bank transfer where money is subtracted from one bank account and then added to another — in precisely that order. The observing process needs exclusivity to avoid other processes making changes that can lead to inconsistencies or surprises. Surprises include inadvertently allowing a transfer from an account that has gone below $0. Isolation guarantees only one process can make a change at a time, and if two are competing for the same data, one of them will have to wait for the other to complete. Embrace Your Laziness Developers need to move quickly with a system they can trust. ACID transactions have been the gold standard of trust within database systems for almost 50 years. Developers work through trade-offs based on requirements, sometimes leading them to work with systems that don’t support ACID transactions. With NoSQL systems, trade-off biases historically tended to tip toward scale and uptime while sacrificing transactions. Bringing ACID transactions to Cassandra has been about reducing trade-offs. Cassandra already has a solid reputation for linear scaling while maintaining uptime in the worst scenarios. Cassandra has been the choice when you need a database that will hold up to what the internet can deliver. Unsurprisingly, the need for transactions has been a source of trade-off conflict for developers. Can We Get a Consensus? In distributed systems, each member node in the larger cluster can act independently or need to coordinate with other nodes. In a transaction where, “Hey, we all need to agree on something,” computer scientists call this consensus, and developing those protocols is a continuous area of improvement. has been a long-established consensus protocol adopted by Cassandra in 2013 for “lightweight transactions.” Paxos Lightweight because it ensures that a single partition data change is isolated in a transaction, but more than one table or partition is not an option. In addition, Paxos requires multiple round trips to gain a consensus, which creates a lot of extra latency and fine print about when to use lightweight transactions in your application. The protocol was developed as the next generation to replace Paxos, and several systems such as Etcd, CockroachDB, and DynamoDB adopted it. It reduced round trips by creating an elected leader. Raft The downside for Cassandra in this approach is that leaders won’t span data centers, so multiple leaders are required (see ). Spanner Having an elected leader also violates the “shared-nothing” principles of Cassandra and would layer new requirements on handling failure. If a node goes down, a new leader has to be elected. Other databases — FaunaDB and FoundationDB, for example — have gone down the route of trying to solve the multileader problem by reducing down to a single, global leader, as described in the . Calvin paper Because these were built for other databases with different requirements, the approaches used in those cases failed to meet the criteria Cassandra expects with failure modes. Cassandra assumes failures as a part of running an extensive distributed system. One or more nodes going offline should not cause rapid performance degradation or availability issues. We needed a different approach. Have We Reached an Accord? We can get very opinionated on what is acceptable for the Cassandra project. Our criteria are about holding true to the core beliefs on how distributed systems should run. Performance and scaling should always be preserved while operating multiple nodes across one or more data centers. We can be pretty demanding, but this is what makes Cassandra the choice for so many organizations. The previous iterations of consensus protocols solved different parts of the problem, but each presented a trade-off that would violate some of Cassandra’s values. It’s been said the next big breakthrough is two papers away from the last. In this case, the paper was , and it took a big swipe at eliminating tradeoffs. Accord Accord addresses two problems that aren’t solved in previous consensus protocols: How can we have a globally available consensus and achieve it in one round trip? The first novel mechanism is the reorder buffer. Assuming commodity hardware is in use, differences in clocks between nodes are inevitable. The reorder buffer measures the difference between nodes in addition to the latency between them. Each replica can use this information to correctly order data from each node and account for the differences, guaranteeing one round-trip consensus with a timestamp protocol. The other mechanism is fast-path electorates. Failure modes can create latency when electing a new leader before resuming. Fast-path electorates use pre-existing features in Cassandra with some novel implementations to maintain a leaderless fast path to quorum under the same level of failure tolerated by Cassandra. More details can be read in the . proposal How Does It Work? The biggest impact will be in developer productivity, so let’s see what that looks like in practice. Consider the following bank account transfer example we mentioned earlier: First is the new syntax you’ll see in the Cassandra Query Language (CQL). Transactions are contained in a and declaration. Everything inside the transaction markers will happen atomically in isolation from other processes. We will transfer $20 from Alice’s account to Bob in this example. It doesn’t get any more classic than that! BEGIN TRANSACTION COMMIT TRANSACTION In section A, we can select data from an existing record and assign the result to a tuple (multiple items stored in a single variable). Depending on how many columns are in the clause, you can store one or more values in the tuple. These values will be used in section B to test conditions before making data changes. SELECT In this case, we will test to see if Alice has $20 in her account before transferring to Bob. If so, then an decrements Alice’s account balance by $20 and then increments Bob’s by $20. If Alice had less than $20, then the changes wouldn’t happen. UPDATE Behind the scenes is a serialized set of database commands that execute exclusively, as seen from the observing process. Across one or more data centers, the transaction only required one round trip to gain consensus, and if any nodes were offline, the action would still occur if at least a quorum of replicas were available. This is exactly how Cassandra likes to work, but we just upped our game with a globally available transaction. What’s Next Accord and all of the work that goes with it is still in progress and slated to be included in the next Cassandra release. Since this is all in open source, those of you that can’t wait can clone a copy of the branch from the Cassandra repository and build your own copy. cep-15-accord For the rest of you, as we get closer to release time, we’ll have builds available for you to use and test. It will be a game-changer for Cassandra, and I’m sure you’ll want to see it for yourself. I’m most interested in hearing from the community about what use cases you’ll find with a globally available transaction running at the speed and resilience you expect from Cassandra. Is it finally time to let go of those last relational database workloads? We’re also anxious to hear your feedback on all our channels, including the Apache Software Foundation . Features in an open source project are constantly evolving to meet users’ needs. That’s why you’ve got a critical role to play in shaping for the future. Slack or project mailing list Apache Cassandra And stay tuned for more use cases and information as we evolve this exciting new feature. You can expect that there will be several talks about this at the digital summit coming up on March 14. You won’t want to miss those. Cassandra Forward By Patrick McFadin, DataStax Patrick McFadin is the co-author of the O’Reilly book 'Managing Cloud Native Data on Kubernetes.' He currently works at DataStax in developer relations and as a contributor to the Apache Cassandra project. Patrick has worked as chief evangelist for Apache Cassandra (he’s also a newly minted Cassandra committer!) and as a consultant for DataStax, where he had a great time building some of the largest deployments in production.

Apache

apachecorp.com

3163 employees

Since 1954

COMPANY RANKING

Apache

EVERGREEN INDEX #9

See more notable links about Apache

Apply to get your Tech Company News Page live on HackerNoon.com today!

Read More Tech Stories Related to #Apache

Apache

Apache

apachecorp.com

3163 employees

Since 1954

COMPANY RANKING

Apache

EVERGREEN INDEX #9

Related CompaniesRank

Apache Story MentionsComplete List

See more notable links about Apache

Apply to get your Tech Company News Page live on HackerNoon.com today!

Read More Tech Stories Related to #Apache

Apache