Data Ingestion Service (DIS)

Data Ingestion Service (DIS)

The Data Ingestion Service (DIS) offers key capabilities to collect, process, and distribute real-time streaming data. DIS docks with a variety of third-party data collection tools and provides a wealth of Agent/SDK. It is suitable for scenarios such as device monitoring, real-time recommendation, and log analysis in IoT, Internet, media and other industries.

The Data Ingestion Service (DIS) offers key capabilities to collect, process, and distribute real-time streaming data. DIS docks with a variety of third-party data collection tools and provides a wealth of Agent/SDK.

Pay per use, starting from $0.0117 USD/hour
Explore DIS:

Product Advantages

Product Advantages

  • Efficient Transfer

    DIS supports millions of concurrent connections and features a millisecond-level response time. Each partition supports data write of up to hundreds of gigabytes per day.
    DIS supports millions of concurrent connections and features a millisecond-level response time. Each partition supports data write of up to hundreds of gigabytes per day.

  • Easy to Use

    You can create DIS streams in seconds and collect data simply by configuring SDKs/agents without programming.
    You can create DIS streams in seconds and collect data simply by configuring SDKs/agents without programming.

  • Secure and Reliable

    DIS uses HTTPS for transmission and controls access permissions by isolating tenants' resources and operations. DIS stores data across AZs to ensure 99.9% availability.
    DIS uses HTTPS for transmission and controls access permissions by isolating tenants' resources and operations. DIS stores data across AZs to ensure 99.9% availability.

  • Simple O&M

    DIS provides managed O&M, allowing you to focus on exploring the value of data. Using DIS reduces costs of Kafka and Flume in building a data transmission system.
    DIS provides managed O&M, allowing you to focus on exploring the value of data. Using DIS reduces costs of Kafka and Flume in building a data transmission system.

Application Scenarios

  • Real-Time Data Processing

  • Real-Time File Transfer

  • Data Intake and Backup

Real-Time Data Processing

Real-Time Data Processing

DIS collects real-time vehicle traffic data and caches the data in DIS streams. The data is analyzed to facilitate traffic control.

Advantages

Real-Time

DIS receives data from application systems in real time.

High Concurrency and Low Latency

Millions of messages can be concurrently processed with an E2E millisecond-level latency.

Openness

DIS can work with various analysis platforms to provide one-stop data transmission and computing.

Related Services

DLI

MRS

Real-Time File Transfer

Real-Time File Transfer

DIS collects files from application systems in real time, uploads them to the cloud, and identifies VIP customers and potential customer needs through analysis and machine learning.

Advantages

Automatic Collection of Incremental Data

Incremental data or new files can be automatically collected.

Easy to Use

DIS provides agents for different platforms to simplify data collection.

High Security

HTTPS-encrypted transmission ensures data security.

Related Services

OBS

MRS

DLI

Data Intake and Backup

Data Intake and Backup

DIS backs up a lot of log files to the cloud for data recovery or exception analysis. Small log files can be merged and dumped into a single large file to accelerate data processing.

Advantages

Multiple Data Sources

DIS supports open source systems such as Kafka and Flume and therefore can easily integrate with other systems.

Efficient Data Transmission

You can scale the DIS stream capacity based on the volume of input data.

Massive Small File Transfer

Small files are migrated to the cloud and then dumped to OBS.

Related Services

OBS

Functions

Functions

  • Data Collection

    DIS provides RESTful APIs, SDKs, and Agent to collect offline data from various sources in real time.
    DIS provides RESTful APIs, SDKs, and Agent to collect offline data from various sources in real time.

  • Data Transmission

    DIS transmits data in real time with high concurrency, high reliability, and low latency.
    DIS transmits data in real time with high concurrency, high reliability, and low latency.

  • Data Movement

    DIS provides multiple connectors to move data freely between storage, computing, and analysis services.
    DIS provides multiple connectors to move data freely between storage, computing, and analysis services.
  • Multiple data sources
    Multiple data sources
    DIS collects data from user application systems, Kafka systems, and Flume systems in real time.
    DIS collects data from user application systems, Kafka systems, and Flume systems in real time.
  • Multiple data intake modes
    Multiple data intake modes
    DIS provides RESTful APIs, SDKs, and agents for real-time data intake with minimum configuration required.
    DIS provides RESTful APIs, SDKs, and agents for real-time data intake with minimum configuration required.
  • DIS supports millions of concurrent connections. Each DIS stream can ingest terabytes of data per day, and each partition supports data write speeds of up to hundreds of gigabytes per day.
    DIS supports millions of concurrent connections. Each DIS stream can ingest terabytes of data per day, and each partition supports data write speeds of up to hundreds of gigabytes per day.
  • The throughput of each DIS stream can scale dynamically by adding partitions.
    The throughput of each DIS stream can scale dynamically by adding partitions.
  • DIS is integrated with OBS, MRS, DWS, CloudTable, DLI, and FunctionGraph to simplify data dumping, computing, and analysis.
    DIS is integrated with OBS, MRS, DWS, CloudTable, DLI, and FunctionGraph to simplify data dumping, computing, and analysis.
  • Data can be imported to cloud services such as CloudTable, DWS, HBASE, RDS, and DCS by simply configuring the agents between DIS and these cloud services.
    Data can be imported to cloud services such as CloudTable, DWS, HBASE, RDS, and DCS by simply configuring the agents between DIS and these cloud services.

  • Data Processing

    DIS comes preloaded with the connectors to extract, aggregate, convert, and preprocess data input from users.
    DIS comes preloaded with the connectors to extract, aggregate, convert, and preprocess data input from users.

  • Data Caching

    DIS preserves real-time data for up to 7 days depending on configurations, ensuring data security.
    DIS preserves real-time data for up to 7 days depending on configurations, ensuring data security.
  • DIS converts data from CSV/JSON format to Parquet/CarbonData format, improving user data quality.
    DIS converts data from CSV/JSON format to Parquet/CarbonData format, improving user data quality.

Usage Guides

Usage Guides

Register with HUAWEI CLOUD to get free services

Register Now