</>
Skip to content
Data Science lessons (6/42)

Data Science — Data Collection

Big data characteristics

  1. Volume: Large amounts
  2. Velocity: Fast generation
  3. Variety: Different types

Hadoop ecosystem

# HDFS
hdfs dfs -put localfile /hdfs/path
hdfs dfs -cat /hdfs/path/file

# MapReduce
hadoop jar hadoop-streaming.jar \
  -mapper mapper.py \
  -reducer reducer.py \
  -input /input \
  -output /output

Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("myapp").getOrCreate()

# Read data
df = spark.read.csv("data.csv", header=True)

# Transformations
df_filtered = df.filter(df["age"] > 25)
df_grouped = df.groupBy("category").count()

# Actions
df_grouped.show()

Data lake vs data warehouse

FeatureData LakeData Warehouse
StructureRawProcessed
SchemaOn readOn write
CostLowerHigher
FlexibilityHighLow

Best practices

  1. Choose right tool
  2. Optimize queries
  3. Monitor performance
  4. Scale as needed

Mini Practice

  1. Process data with Spark
  2. Query with HDFS
  3. Build ETL pipeline
  4. Optimize performance

Up Next

Continue with Spark - Distributed computing.

Related Topics

Data TypesData CleaningData PreparationExploratory Data Analysis

Frequently Asked Questions about Data Collection

What is Data Collection in Data Science?

Data Collection is a fundamental concept in Data Science. This lesson explains it step by step with clear examples, making it easy for beginners to understand.

How do I learn Data Collection?

Start by reading the explanation above, then try the code examples. Practice by modifying the examples and experimenting with different values. Hands-on practice is the best way to learn Data Collection.

Why is Data Collection important in Data Science?

Data Collection is essential for Data Science development. Understanding this concept will help you write better code and solve real-world problems more effectively.