Hadoop

Setup

If ~/hadoop/ doesn't exist or is empty, read setup.md and start the conversation naturally.

When to Use

User works with Hadoop ecosystem (HDFS, YARN, MapReduce, Hive). Agent handles cluster diagnostics, job optimization, storage management, and troubleshooting distributed processing failures.

Architecture

Memory lives in ~/hadoop/. See memory-template.md for structure.

~/hadoop/
├── memory.md        # Cluster configs, common issues, preferences
├── clusters/        # Per-cluster notes and configs
│   └── {name}.md    # Specific cluster context
└── scripts/         # Custom diagnostic scripts

Quick Reference

Core Rules

1. Verify Cluster State First

Before any operation, check cluster health:

hdfs dfsadmin -report
yarn node -list

Never assume cluster is healthy. A single dead DataNode changes everything.

2. Storage Before Compute

HDFS issues cascade into job failures. Always check:

hdfs dfs -df -h                    # Capacity
hdfs fsck / -files -blocks         # Block health

A job failing with "No space left" is storage, not code.

3. Resource Calculator Awareness

YARN allocates based on configured scheduler. Know which is active:

yarn rmadmin -getServiceState rm1
cat /etc/hadoop/conf/yarn-site.xml | grep scheduler

Default (Capacity) vs Fair scheduler behave very differently.

4. Replication Factor Context

Default replication=3. For temp data, suggest 1-2 to save space:

hdfs dfs -setrep -w 1 /tmp/scratch/

For critical data, verify replication is honored:

hdfs fsck /data/critical -files -blocks -replicaDetails

5. Log Location Awareness

Hadoop logs scatter across machines. Key locations:

6. Safe Mode Handling

NameNode enters safe mode on startup or low block count:

hdfs dfsadmin -safemode get        # Check status
hdfs dfsadmin -safemode leave      # Exit (if blocks OK)

Never force-leave if blocks are actually missing.

7. Memory Settings Matter

90% of "job killed" issues are memory:

# Container settings
yarn.nodemanager.resource.memory-mb     # Total per node
yarn.scheduler.minimum-allocation-mb    # Min container
mapreduce.map.memory.mb                 # Map task
mapreduce.reduce.memory.mb              # Reduce task

Check these before assuming code is wrong.

HDFS Operations