具体描述
多位专家联袂推荐，360大数据专家撰写，基于Spark 2.1.0剖析架构与实现精髓。细化到方法级，提炼出多个流程图，立体呈现架构、环境、调度、存储、计算、部署、API七大核心设计。本书一共有10章内容，主要包括以下部分。
准备部分（第1～2章）：简单介绍了Spark的环境搭建和基本原理。本部分通过详尽的描述，有效降低了读者进入Spark世界的门槛，同时能对Spark背景知识及整体设计有宏观的认识。
基础部分（第3～5章）：介绍Spark的基础设施（包括配置、RPC、度量等）、SparkContext的初始化、Spark执行所需要的环境等内容。经过此部分的学习，将能够对RPC框架的设计、执行环境的功能有深入的理解，这也是对核心内容了解的前提。
核心部分（第6～9章）：为Spark最核心的部分，包括存储体系、调度系统、计算引擎、部署模式等。通过本部分的学习，读者将充分了解Spark的数据处理体系细节，能够对Spark核心功能进行扩展、性能优化以及对线上问题进行精准排查。
API部分（第10章）：这部分主要对Spark的新老API进行对比，对新API进行简单介绍。
《大数据架构师的实战指南：从原理到应用》 内容简介 在这个数据爆炸的时代，掌握高效处理海量数据的能力已成为每个技术从业者的核心竞争力。本书并非一本纯粹的理论书籍，而是以实际应用为导向，深入浅出地剖析了构建和优化现代大数据处理系统所需的关键技术和核心理念。我们将带领读者从根本上理解大数据架构的设计原则，并一步步构建出稳定、高效、可扩展的数据处理流水线。 第一部分：大数据时代的机遇与挑战 在信息技术飞速发展的今天，我们正经历着一场前所未有的数据革命。社交媒体、物联网设备、电商交易、科学研究，无一不在产生着海量的数据。这些数据中蕴藏着巨大的商业价值和深刻的洞察力，能够帮助企业做出更明智的决策，优化运营，甚至催生全新的商业模式。然而，与此同时，处理这些海量数据也带来了前所未有的挑战： 数据规模的挑战： 传统的数据存储和处理技术在面对PB甚至EB级别的数据时显得力不从心。如何高效地存储、管理和访问如此庞大的数据集是首要解决的问题。 数据多样性的挑战： 数据来源广泛，格式各异，既有结构化的关系型数据，也有半结构化的JSON、XML，更有海量的非结构化文本、图像、音视频等。如何统一有效地处理这些异构数据是另一大难题。 数据处理速度的挑战： 实时数据分析、即时响应等需求对数据处理的实时性提出了极高的要求。传统的批处理模式往往难以满足低延迟的需求。 数据质量与安全挑战： 数据的准确性、完整性、一致性直接影响着分析结果的可靠性。同时，如何保护敏感数据，防止数据泄露，确保数据安全合规，也是至关重要的考量。 本书的第一部分旨在为读者构建一个宏观的认知框架，让大家深刻理解大数据带来的机遇，以及在拥抱这些机遇时必须直面的严峻挑战。我们将回顾大数据技术的发展历程，梳理不同阶段的关键技术演进，并重点分析当前大数据架构面临的共性问题，为后续深入的技术探讨奠定基础。 第二部分：核心分布式计算引擎剖析 理解大数据处理的核心，离不开对分布式计算引擎的深入洞察。本书将重点剖析当前业界广泛应用且技术成熟的两大分布式计算引擎：MapReduce 和 Spark。 MapReduce：批处理的基石 原理与设计思想： 我们将追溯MapReduce的起源，深入理解其“Map”和“Reduce”两个核心操作的设计理念。详细讲解数据如何通过Map阶段并行处理，然后经过Shuffle（混洗）过程汇聚到Reduce阶段进行聚合，最终产生结果。 执行流程与数据流： 通过丰富的图示和案例，清晰地展现MapReduce作业的生命周期，包括任务的提交、调度、执行、容错机制（如任务重试、数据冗余）等。 优缺点与适用场景： 分析MapReduce在批处理方面的优势，如稳定可靠、易于理解；同时也会指出其在迭代计算、交互式查询等方面的不足，例如高延迟、磁盘I/O开销大等。 Spark：内存计算的革命 Spark核心概念： 引入Spark的RDD（弹性分布式数据集）作为其核心抽象，讲解RDD的惰性计算、不可变性、宽依赖与窄依赖等特性。 Spark的执行模型： 深入解析Spark的DAG（有向无环图）调度器，理解Spark如何将作业转化为DAG，并进行有效的任务调度和优化。 Spark的内存计算优势： 重点阐述Spark如何通过将中间计算结果缓存到内存中，显著减少磁盘I/O，从而大幅提升数据处理速度，尤其是在迭代算法和交互式分析场景下。 Spark的四大组件： 详细介绍Spark Streaming（实时流处理）、Spark SQL（结构化数据处理）、MLlib（机器学习库）和GraphX（图计算）等组件，展示Spark在不同应用场景下的强大能力。 Spark的优缺点与应用： 总结Spark在处理速度、易用性、功能丰富性方面的优势，并探讨其在内存消耗、任务调度复杂性等方面的权衡。 通过对这两个引擎的深入对比分析，读者将能够清晰地认识到它们各自的设计哲学、技术优势以及最适合的应用场景，从而在面对具体项目需求时，做出最恰当的技术选型。 第三部分：大数据存储与管理系统 海量数据的存储和高效访问是大数据处理的另一重要支柱。本书将重点介绍当前主流的分布式存储系统，它们是承载和管理海量数据的基石。 HDFS（Hadoop Distributed File System）： 架构设计： 深入讲解HDFS的NameNode（元数据管理）和DataNode（数据存储）的职责，以及Secondary NameNode的作用。 数据读写流程： 详细剖析客户端如何向HDFS读写数据，包括文件分割、块（Block）概念、副本（Replication）机制、写入流程（DataOutputStream、DFSOutputStream）、读取流程（DFSInputStream）等。 容错机制： 讲解HDFS如何通过副本机制、心跳检测、故障转移等策略来保证数据的高可用性和持久性。 适用场景： 分析HDFS作为海量大数据文件存储的优势，以及其在延迟敏感型应用上的局限性。 NoSQL数据库： NoSQL的崛起与分类： 介绍NoSQL数据库产生的背景，以及与传统关系型数据库的根本区别。我们将重点分类介绍几种代表性的NoSQL数据库类型： 键值存储（Key-Value Stores）： 如Redis、Memcached，讲解其极高的读写性能，以及适合缓存、会话管理等场景。 列族存储（Column-Family Stores）： 如HBase、Cassandra，深入理解其按列存储的特点，以及在大规模数据写入和范围查询方面的优势，适合日志、时序数据等。 文档数据库（Document Databases）： 如MongoDB，介绍其以JSON/BSON格式存储数据，灵活的Schema，以及适合内容管理、用户画像等场景。 图数据库（Graph Databases）： 如Neo4j，讲解其专门用于存储和查询节点与关系的网络数据，适合社交网络、推荐系统等。 CAP理论与一致性模型： 结合CAP理论，解释不同NoSQL数据库在一致性（Consistency）、可用性（Availability）和分区容错性（Partition Tolerance）之间的权衡，帮助读者理解其设计哲学。 数据仓库与数据湖： 数据仓库的演进： 回顾数据仓库的概念，以及其在结构化数据分析方面的作用。 数据湖的兴起： 介绍数据湖作为统一存储海量原始数据的概念，它可以存储结构化、半结构化和非结构化数据，并提供灵活的访问和处理方式。 数据仓库与数据湖的对比与结合： 探讨两者在数据处理模式、Schema灵活性、存储成本等方面的差异，以及如何通过“数据湖仓一体”等架构实现两者的优势互补。 第四部分：构建高效可靠的大数据流水线 拥有了强大的计算引擎和存储系统，我们还需要将它们有效地组织起来，构建出端到端的大数据处理流水线。本部分将聚焦于大数据流水线的关键环节和技术。 数据采集与接入： 批量数据采集： 介绍Sqoop等工具在关系型数据库与HDFS之间进行数据同步的原理和实践。 实时数据采集： 重点讲解Apache Kafka等分布式消息队列的架构和应用，理解其作为数据缓冲、解耦生产者与消费者的核心作用。分析Kafka的Topic、Partition、Producer、Consumer模型，以及其高吞吐量、低延迟的特点。 日志收集： 介绍Flume等工具在收集、聚合和传输日志数据的应用。 数据清洗与预处理： ETL（Extract, Transform, Load）概念： 解释ETL在数据集成和准备中的重要性。 利用Spark进行数据清洗： 通过具体的Spark代码示例，演示如何使用DataFrame/Dataset API进行数据过滤、转换、聚合、去重、缺失值处理等操作，确保数据质量。 数据质量监控： 讨论如何建立数据质量检测机制，及时发现和处理数据异常。 数据分析与挖掘： 批处理分析： 结合Spark SQL和MapReduce，演示如何进行复杂的SQL查询、统计分析、报表生成等。 实时流处理： 深入讲解Spark Streaming或Flink等流处理框架，展示如何对Kafka等数据源进行实时数据分析、事件检测、告警等。 机器学习与图计算： 结合Spark MLlib和GraphX，介绍如何构建和部署机器学习模型，以及进行图数据的分析，例如社交网络分析、关联规则挖掘等。 数据可视化与洞察： BI工具集成： 介绍如何将大数据平台与Tableau、Power BI等商业智能工具连接，实现数据的可视化呈现。 前端可视化库： 简要介绍D3.js、ECharts等前端可视化库，展示如何构建自定义的交互式数据图表。 第五部分：大数据架构的实践考量与未来趋势 在掌握了核心技术和构建方法后，本书的最后部分将回归到更宏观的实践层面，并展望大数据技术的未来发展。 集群管理与资源调度： YARN（Yet Another Resource Negotiator）： 深入理解YARN的 ResourceManager 和 NodeManager 架构，以及其在资源分配、任务调度、应用管理方面的作用。 Kubernetes在大数据中的应用： 探讨Kubernetes作为容器编排平台，如何为大数据应用提供更灵活、更高效的部署、管理和扩展能力。 数据安全与隐私保护： 访问控制与认证： 讨论Kerberos等认证机制在保障大数据平台安全访问方面的作用。 数据加密： 介绍静态数据加密和传输中数据加密的重要性。 合规性要求： 提及GDPR等法规对数据处理的合规性要求，以及如何在架构设计中予以考虑。 性能优化策略： 代码层面优化： 讲解RDD/DataFrame的性能调优技巧，如广播变量、累加器、缓存策略、避免shuffle等。 参数调优： 讨论HDFS、Spark等组件的关键参数配置，以达到最佳性能。 硬件选择与网络优化： 简要分析硬件配置、网络带宽对大数据性能的影响。 云原生大数据架构： 公有云大数据服务： 介绍AWS EMR、Azure HDInsight、Google Cloud Dataproc等云厂商提供的大数据服务，以及其带来的弹性、成本效益和易用性。 Serverless大数据处理： 展望Serverless计算在未来大数据处理中的潜力。 大数据技术发展趋势： 数据湖仓一体化： 探讨如何实现数据湖和数据仓库的融合，提供更统一、更高效的数据服务。 实时数仓与流批一体： 分析实时数据处理与批处理的融合趋势。 AI与大数据的结合： 展望AI在数据治理、自动化运维、智能分析等方面的应用。 本书旨在为读者提供一个全面、系统、深入的大数据架构设计与实践的学习路径。通过理论与实践相结合，引导读者掌握构建强大、灵活、可扩展的大数据处理能力，从而在数据驱动的时代抓住机遇，应对挑战，成为一名出色的 \ 大数据架构师 \。
作者简介
耿嘉安
10余年IT行业相关经验。先后就职于阿里巴巴、艺龙、360，专注于开源和大数据领域。在大量的工作实践中，对J2EE、JVM、Tomcat、Spring、Hadoop、Spark、MySQL、Redis都有深入研究，尤其喜欢剖析开源项目的源码实现。早期从事J2EE企业级应用开发，对Java相关技术有独到见解。著有《深入理解Spark：核心思想与源码分析》一书。
目录信息
目录 Contents
本书赞誉
前言
第1章 环境准备 ········································1
1.1 运行环境准备 ···········································2
1.1.1 安装JDK ·········································2
1.1.2 安装Scala ········································2
1.1.3 安装Spark ·······································3
1.2 Spark初体验 ···································4
1.2.1 运行spark-shell ·······························4
1.2.2 执行word count ······························5
1.2.3 剖析spark-shell ·······························9
1.3 阅读环境准备 ·········································14
1.3.1 安装SBT ·······································15
1.3.2 安装Git ·········································15
1.3.3 安装Eclipse Scala IDE插件 ········15
1.4 Spark源码编译与调试 ·························17
1.5 小结 ···························23
第2章 设计理念与基本架构 ···············24
2.1 初识Spark ··································25
2.1.1 Hadoop MRv1的局限···················25
2.1.2 Spark的特点 ·································26
2.1.3 Spark使用场景 ·····························28
2.2 Spark基础知识 ······································29
2.3 Spark基本设计思想 ·····························31
2.3.1 Spark模块设计 ·····························32
2.3.2 Spark模型设计 ·····························34
2.4 Spark基本架构 ···································36
2.5 小结 ·································38
第3章 Spark基础设施 ·························39
3.1 Spark配置 ········································40
3.1.1 系统属性中的配置 ·······················40
3.1.2 使用SparkConf配置的API ·········41
3.1.3 克隆SparkConf配置 ····················42
3.2 Spark内置RPC框架 ····························42
3.2.1 RPC配置TransportConf ··············45
3.2.2 RPC客户端工厂Transport- ClientFactory ·······················47
3.2.3 RPC服务端TransportServer ········53
3.2.4 管道初始化 ···································56
3.2.5 TransportChannelHandler详解 ·····57
3.2.6 服务端RpcHandler详解 ··············63
3.2.7 服务端引导程序Transport-ServerBootstrap ·····················68
3.2.8 客户端TransportClient详解 ········71
3.3 事件总线 ····································78
3.3.1 ListenerBus的继承体系 ···············79
3.3.2 SparkListenerBus详解 ··················80
3.3.3 LiveListenerBus详解 ····················83
3.4 度量系统 ···········································87
3.4.1 Source继承体系 ···························87
3.4.2 Sink继承体系 ·······························89
3.5 小结 ·········································92
第4章 SparkContext的初始化 ·········93
4.1 SparkContext概述 ·································94
4.2 创建Spark环境 ·····································97
4.3 SparkUI的实现 ····································100
4.3.1 SparkUI概述 ·······························100
4.3.2 WebUI框架体系 ·························102
4.3.3 创建SparkUI ·······························107
4.4 创建心跳接收器 ··································111
4.5 创建和启动调度系统··························112
4.6 初始化块管理器BlockManager ·······114
4.7 启动度量系统 ·······························114
4.8 创建事件日志监听器··························115
4.9 创建和启动ExecutorAllocation-Manager ··························116
4.10 ContextCleaner的创建与启动 ········120
4.10.1 创建ContextCleaner ·················120
4.10.2 启动ContextCleaner ·················120
4.11 额外的SparkListener与启动事件总线 ··························122
4.12 Spark环境更新 ··································123
4.13 SparkContext初始化的收尾 ···········127
4.14 SparkContext提供的常用方法 ·······128
4.15 SparkContext的伴生对象················130
4.16 小结 ····································131
第5章 Spark执行环境 ························132
5.1 SparkEnv概述 ·································133
5.2 安全管理器SecurityManager ············133
5.3 RPC环境 ·········································135
5.3.1 RPC端点RpcEndpoint ···············136
5.3.2 RPC端点引用RpcEndpointRef ···139
5.3.3 创建传输上下文TransportConf ···142
5.3.4 消息调度器Dispatcher ···············142
5.3.5 创建传输上下文Transport-Context ·························154
5.3.6 创建传输客户端工厂Transport-ClientFactory ····················159
5.3.7 创建TransportServer ···················160
5.3.8 客户端请求发送 ·························162
5.3.9 NettyRpcEnv中的常用方法 ·······173
5.4 序列化管理器SerializerManager ·····175
5.5 广播管理器BroadcastManager ·········178
5.6 map任务输出跟踪器 ··························185
5.6.1 MapOutputTracker的实现 ··········187
5.6.2 MapOutputTrackerMaster的实现原理 ·······················191
5.7 构建存储体系 ·······································199
5.8 创建度量系统 ·······································201
5.8.1 MetricsCon?g详解 ·····················203
5.8.2 MetricsSystem中的常用方法 ····207
5.8.3 启动MetricsSystem ····················209
5.9 输出提交协调器 ··································211
5.9.1 OutputCommitCoordinator-Endpoint的实现 ··················211
5.9.2 OutputCommitCoordinator的实现 ··························212
5.9.3 OutputCommitCoordinator的工作原理 ························216
5.10 创建SparkEnv ····································217
5.11 小结 ·····································217
第6章 存储体系 ·····································219
6.1 存储体系概述 ·······································220
6.1.1 存储体系架构 ·····························220
6.1.2 基本概念 ·····································222
6.2 Block信息管理器 ································227
6.2.1 Block锁的基本概念 ···················227
6.2.2 Block锁的实现 ···························229
6.3 磁盘Block管理器 ······························234
6.3.1 本地目录结构 ·····························234
6.3.2 DiskBlockManager提供的方法 ···························236
6.4 磁盘存储DiskStore ·····························239
6.5 内存管理器 ·····································242
6.5.1 内存池模型 ·································243
6.5.2 StorageMemoryPool详解 ···········244
6.5.3 MemoryManager模型 ················247
6.5.4 Uni?edMemoryManager详解 ····250
6.6 内存存储MemoryStore ······················252
6.6.1 MemoryStore的内存模型 ··········253
6.6.2 MemoryStore提供的方法 ··········255
6.7 块管理器BlockManager ····················265
6.7.1 BlockManager的初始化 ·············265
6.7.2 BlockManager提供的方法 ·········266
6.8 BlockManagerMaster对Block-Manager的管理 ·················285
6.8.1 BlockManagerMaster的职责 ······285
6.8.2 BlockManagerMasterEndpoint详解 ·································286
6.8.3 BlockManagerSlaveEndpoint详解 ·····························289
6.9 Block传输服务 ····································290
6.9.1 初始化NettyBlockTransfer-Service ···························291
6.9.2 NettyBlockRpcServer详解 ·········292
6.9.3 Shuf?e客户端 ·····························296
6.10 DiskBlockObjectWriter详解 ···········305
6.11 小结 ·······································308
第7章 调度系统 ·····································309
7.1 调度系统概述 ·······································310
7.2 RDD详解 ·····································312
7.2.1 为什么需要RDD ························312
7.2.2 RDD实现的初次分析 ················313
7.2.3 RDD依赖 ····································316
7.2.4 分区计算器Partitioner················318
7.2.5 RDDInfo ······································320
7.3 Stage详解 ········································321
7.3.1 ResultStage的实现 ·····················322
7.3.2 Shuf?eMapStage的实现 ·············323
7.3.3 StageInfo ······································324
7.4 面向DAG的调度器DAGScheduler ···326
7.4.1 JobListener与JobWaiter ·············326
7.4.2 ActiveJob详解 ····························328
7.4.3 DAGSchedulerEventProcessLoop的简要介绍 ·······················328
7.4.4 DAGScheduler的组成 ················329
7.4.5 DAGScheduler提供的常用方法 ···330
7.4.6 DAGScheduler与Job的提交 ····334
7.4.7 构建Stage····································337
7.4.8 提交ResultStage ························341
7.4.9 提交还未计算的Task ·················343
7.4.10 DAGScheduler的调度流程 ······347
7.4.11 Task执行结果的处理 ··············348
7.5 调度池Pool ······································351
7.5.1 调度算法 ·······························352
7.5.2 Pool的实现 ·································354
7.5.3 调度池构建器 ·····························357
7.6 任务集合管理器TaskSetManager ···363
7.6.1 Task集合 ·····································363
7.6.2 TaskSetManager的成员属性 ······364
7.6.3 调度池与推断执行 ·····················366
7.6.4 Task本地性 ·································370
7.6.5 TaskSetManager的常用方法 ······373
7.7 运行器后端接口LauncherBackend ···383
7.7.1 BackendConnection的实现 ········384
7.7.2 LauncherBackend的实现 ···········386
7.8 调度后端接口SchedulerBackend ····389
7.8.1 SchedulerBackend的定义 ··········389
7.8.2 LocalSchedulerBackend的实现分析 ································390
7.9 任务结果获取器TaskResultGetter ···394
7.9.1 处理成功的Task ·························394
7.9.2 处理失败的Task ·························396
7.10 任务调度器TaskScheduler ··············397
7.10.1 TaskSchedulerImpl的属性 ·····397
7.10.2 TaskSchedulerImpl的初始化 ···399
7.10.3 TaskSchedulerImpl的启动 ·····399
7.10.4 TaskSchedulerImpl与Task的提交 ·······················400
7.10.5 TaskSchedulerImpl与资源分配 ···························402
7.10.6 TaskSchedulerImpl的调度流程 ······························405
7.10.7 TaskSchedulerImpl对执行结果的处理 ·····························406
7.10.8 TaskSchedulerImpl的常用方法 ···409
7.11 小结 ·······································412
第8章 计算引擎 ·····································413
8.1 计算引擎概述 ·······································414
8.2 内存管理器与执行内存 ·····················417
8.2.1 ExecutionMemoryPool详解 ·······417
8.2.2 MemoryManager模型与执行内存 ··························420
8.2.3 Uni?edMemoryManager与执行内存 ·······················421
8.3 内存管理器与Tungsten ·····················423
8.3.1 MemoryBlock详解 ·····················423
8.3.2 MemoryManager模型与Tungsten ···························425
8.3.3 Tungsten的内存分配器 ··············425
8.4 任务内存管理器 ··································431
8.4.1 TaskMemoryManager详解 ·········431
8.4.2 内存消费者 ·······················439
8.4.3 执行内存整体架构 ·····················441
8.5 Task详解 ······································443
8.5.1 任务上下文TaskContext ············443
8.5.2 Task的定义 ·································446
8.5.3 Shuf?eMapTask的实现 ··············449
8.5.4 ResultTask的实现 ·······················450
8.6 IndexShuf?eBlockResolver详解 ······451
8.7 采样与估算 ···········································455
8.7.1 SizeTracker的实现分析 ·············455
8.7.2 SizeTracker的工作原理 ·············457
8.8 特质WritablePartitionedPair- Collection ······················458
8.9 AppendOnlyMap的实现分析 ···········460
8.9.1 AppendOnlyMap的容量增长 ····461
8.9.2 AppendOnlyMap的数据更新 ····462
8.9.3 AppendOnlyMap的缓存聚合算法 ·····························464
8.9.4 AppendOnlyMap的内置排序 ····466
8.9.5 AppendOnlyMap的扩展 ············467
8.10 PartitionedPairBuffer的实现分析 ···469
8.10.1 PartitionedPairBuffer的容量增长 ······················469
8.10.2 PartitionedPairBuffer的插入 ···470
8.10.3 PartitionedPairBuffer的迭代器 ···471
8.11 外部排序器 ·········································472
8.11.1 ExternalSorter详解 ·················473
8.11.2 Shuf?eExternalSorter详解 ······487
8.12 Shuf?e管理器 ····································490
8.12.1 Shuf?eWriter详解 ··················491
8.12.2 Shuf?eBlockFetcherIterator详解 ······························502
8.12.3 BlockStoreShuf?eReader详解 ···510
8.12.4 SortShuf?eManager详解 ········513
8.13 map端与reduce端的Shuf?e组合 ······························516
8.14 小结 ·········································519
第9章 部署模式 ········································520
9.1 心跳接收器HeartbeatReceiver ·········521
9.2 Executor的实现分析 ··························527
9.2.1 Executor的心跳报告 ··················528
9.2.2 运行Task ·····································530
9.3 local部署模式 ······································535
9.4 持久化引擎PersistenceEngine ··········537
9.4.1 基于文件系统的持久化引擎 ·····539
9.4.2 基于ZooKeeper的持久化引擎 ···541
9.5 领导选举代理 ·······································542
9.6 Master详解 ···········································546
9.6.1 启动Master ·································549
9.6.2 检查Worker超时························553
9.6.3 被选举为领导时的处理 ·············554
9.6.4 一级资源调度 ·····························558
9.6.5 注册Worker·································568
9.6.6 更新Worker的最新状态············570
9.6.7 处理Worker的心跳····················570
9.6.8 注册Application··························571
9.6.9 处理Executor的申请 ·················573
9.6.10 处理Executor的状态变化 ·······573
9.6.11 Master的常用方法 ···················574
9.7 Worker详解 ································578
9.7.1 启动Worker·································581
9.7.2 向Master注册Worker ···············584
9.7.3 向Master发送心跳 ····················589
9.7.4 Worker与领导选举·····················591
9.7.5 运行Driver ··································593
9.7.6 运行Executor ······························594
9.7.7 处理Executor的状态变化 ·········599
9.8 StandaloneAppClient实现 ·················600
9.8.1 ClientEndpoint的实现分析 ········601
9.8.2 StandaloneAppClient的实现分析 ······························606
9.9 StandaloneSchedulerBackend的实现分析 ························607
9.9.1 StandaloneSchedulerBackend的属性 ····························607
9.9.2 DriverEndpoint的实现分析 ·······609
9.9.3 StandaloneSchedulerBackend的启动 ··························614
9.9.4 StandaloneSchedulerBackend的停止 ·························617
9.9.5 StandaloneSchedulerBackend与资源分配 ················618
9.10 CoarseGrainedExecutorBackend详解 ····························619
9.10.1 CoarseGrainedExecutorBackend进程 ··························620
9.10.2 CoarseGrainedExecutorBackend的功能分析 ·························622
9.11 local-cluster部署模式 ·······················625
9.11.1 启动本地集群 ····························625
9.11.2 local-cluster部署模式的启动过程 ·································627
9.11.3 local-cluster部署模式下Executor的分配过程 ·················628
9.11.4 local-cluster部署模式下的任务提交执行过程 ····························629
9.12 Standalone部署模式 ·························631
9.12.1 Standalone部署模式的启动过程 ························632
9.12.2 Standalone部署模式下Executor的分配过程 ················634
9.12.3 Standalone部署模式的资源回收 ·····························635
9.12.4 Standalone部署模式的容错机制 ······························636
9.13 其他部署方案 ·····································639
9.13.1 YARN·········································639
9.13.2 Mesos ·········································644
9.14 小结 ·······································646
第10章 Spark API ································647
10.1 基本概念·····································648
10.2 数据源DataSource ····························650
10.2.1 DataSourceRegister详解 ··········650
10.2.2 DataSource详解 ························651
10.3 检查点的实现 ···································655
10.3.1 CheckpointRDD的实现············655
10.3.2 RDDCheckpointData的实现 ····660
10.3.3 ReliableRDDCheckpointData的实现 ························662
10.4 RDD的再次分析 ·······························663
10.4.1 转换API ····································663
10.4.2 动作API ····································665
10.4.3 检查点API的实现分析 ···········667
10.4.4 迭代计算 ···································669
10.5 数据集合Dataset ·······························671
10.6 DataFrameReader详解 ·····················673
10.7 SparkSession详解 ·····························676
10.7.1 SparkSession的构建器Builder ···676
10.7.2 SparkSession的API ·················679
10.8 word count例子 ·································679
10.8.1 Job准备阶段 ·····························680
10.8.2 Job的提交与调度 ·····················685
10.9 小结 ········································689
附录 ···········································690
· · · · · · (收起)
本书赞誉
前言
第1章 环境准备 ········································1
1.1 运行环境准备 ···········································2
1.1.1 安装JDK ·········································2
1.1.2 安装Scala ········································2
1.1.3 安装Spark ·······································3
1.2 Spark初体验 ···································4
1.2.1 运行spark-shell ·······························4
1.2.2 执行word count ······························5
1.2.3 剖析spark-shell ·······························9
1.3 阅读环境准备 ·········································14
1.3.1 安装SBT ·······································15
1.3.2 安装Git ·········································15
1.3.3 安装Eclipse Scala IDE插件 ········15
1.4 Spark源码编译与调试 ·························17
1.5 小结 ···························23
第2章 设计理念与基本架构 ···············24
2.1 初识Spark ··································25
2.1.1 Hadoop MRv1的局限···················25
2.1.2 Spark的特点 ·································26
2.1.3 Spark使用场景 ·····························28
2.2 Spark基础知识 ······································29
2.3 Spark基本设计思想 ·····························31
2.3.1 Spark模块设计 ·····························32
2.3.2 Spark模型设计 ·····························34
2.4 Spark基本架构 ···································36
2.5 小结 ·································38
第3章 Spark基础设施 ·························39
3.1 Spark配置 ········································40
3.1.1 系统属性中的配置 ·······················40
3.1.2 使用SparkConf配置的API ·········41
3.1.3 克隆SparkConf配置 ····················42
3.2 Spark内置RPC框架 ····························42
3.2.1 RPC配置TransportConf ··············45
3.2.2 RPC客户端工厂Transport- ClientFactory ·······················47
3.2.3 RPC服务端TransportServer ········53
3.2.4 管道初始化 ···································56
3.2.5 TransportChannelHandler详解 ·····57
3.2.6 服务端RpcHandler详解 ··············63
3.2.7 服务端引导程序Transport-ServerBootstrap ·····················68
3.2.8 客户端TransportClient详解 ········71
3.3 事件总线 ····································78
3.3.1 ListenerBus的继承体系 ···············79
3.3.2 SparkListenerBus详解 ··················80
3.3.3 LiveListenerBus详解 ····················83
3.4 度量系统 ···········································87
3.4.1 Source继承体系 ···························87
3.4.2 Sink继承体系 ·······························89
3.5 小结 ·········································92
第4章 SparkContext的初始化 ·········93
4.1 SparkContext概述 ·································94
4.2 创建Spark环境 ·····································97
4.3 SparkUI的实现 ····································100
4.3.1 SparkUI概述 ·······························100
4.3.2 WebUI框架体系 ·························102
4.3.3 创建SparkUI ·······························107
4.4 创建心跳接收器 ··································111
4.5 创建和启动调度系统··························112
4.6 初始化块管理器BlockManager ·······114
4.7 启动度量系统 ·······························114
4.8 创建事件日志监听器··························115
4.9 创建和启动ExecutorAllocation-Manager ··························116
4.10 ContextCleaner的创建与启动 ········120
4.10.1 创建ContextCleaner ·················120
4.10.2 启动ContextCleaner ·················120
4.11 额外的SparkListener与启动事件总线 ··························122
4.12 Spark环境更新 ··································123
4.13 SparkContext初始化的收尾 ···········127
4.14 SparkContext提供的常用方法 ·······128
4.15 SparkContext的伴生对象················130
4.16 小结 ····································131
第5章 Spark执行环境 ························132
5.1 SparkEnv概述 ·································133
5.2 安全管理器SecurityManager ············133
5.3 RPC环境 ·········································135
5.3.1 RPC端点RpcEndpoint ···············136
5.3.2 RPC端点引用RpcEndpointRef ···139
5.3.3 创建传输上下文TransportConf ···142
5.3.4 消息调度器Dispatcher ···············142
5.3.5 创建传输上下文Transport-Context ·························154
5.3.6 创建传输客户端工厂Transport-ClientFactory ····················159
5.3.7 创建TransportServer ···················160
5.3.8 客户端请求发送 ·························162
5.3.9 NettyRpcEnv中的常用方法 ·······173
5.4 序列化管理器SerializerManager ·····175
5.5 广播管理器BroadcastManager ·········178
5.6 map任务输出跟踪器 ··························185
5.6.1 MapOutputTracker的实现 ··········187
5.6.2 MapOutputTrackerMaster的实现原理 ·······················191
5.7 构建存储体系 ·······································199
5.8 创建度量系统 ·······································201
5.8.1 MetricsCon?g详解 ·····················203
5.8.2 MetricsSystem中的常用方法 ····207
5.8.3 启动MetricsSystem ····················209
5.9 输出提交协调器 ··································211
5.9.1 OutputCommitCoordinator-Endpoint的实现 ··················211
5.9.2 OutputCommitCoordinator的实现 ··························212
5.9.3 OutputCommitCoordinator的工作原理 ························216
5.10 创建SparkEnv ····································217
5.11 小结 ·····································217
第6章 存储体系 ·····································219
6.1 存储体系概述 ·······································220
6.1.1 存储体系架构 ·····························220
6.1.2 基本概念 ·····································222
6.2 Block信息管理器 ································227
6.2.1 Block锁的基本概念 ···················227
6.2.2 Block锁的实现 ···························229
6.3 磁盘Block管理器 ······························234
6.3.1 本地目录结构 ·····························234
6.3.2 DiskBlockManager提供的方法 ···························236
6.4 磁盘存储DiskStore ·····························239
6.5 内存管理器 ·····································242
6.5.1 内存池模型 ·································243
6.5.2 StorageMemoryPool详解 ···········244
6.5.3 MemoryManager模型 ················247
6.5.4 Uni?edMemoryManager详解 ····250
6.6 内存存储MemoryStore ······················252
6.6.1 MemoryStore的内存模型 ··········253
6.6.2 MemoryStore提供的方法 ··········255
6.7 块管理器BlockManager ····················265
6.7.1 BlockManager的初始化 ·············265
6.7.2 BlockManager提供的方法 ·········266
6.8 BlockManagerMaster对Block-Manager的管理 ·················285
6.8.1 BlockManagerMaster的职责 ······285
6.8.2 BlockManagerMasterEndpoint详解 ·································286
6.8.3 BlockManagerSlaveEndpoint详解 ·····························289
6.9 Block传输服务 ····································290
6.9.1 初始化NettyBlockTransfer-Service ···························291
6.9.2 NettyBlockRpcServer详解 ·········292
6.9.3 Shuf?e客户端 ·····························296
6.10 DiskBlockObjectWriter详解 ···········305
6.11 小结 ·······································308
第7章 调度系统 ·····································309
7.1 调度系统概述 ·······································310
7.2 RDD详解 ·····································312
7.2.1 为什么需要RDD ························312
7.2.2 RDD实现的初次分析 ················313
7.2.3 RDD依赖 ····································316
7.2.4 分区计算器Partitioner················318
7.2.5 RDDInfo ······································320
7.3 Stage详解 ········································321
7.3.1 ResultStage的实现 ·····················322
7.3.2 Shuf?eMapStage的实现 ·············323
7.3.3 StageInfo ······································324
7.4 面向DAG的调度器DAGScheduler ···326
7.4.1 JobListener与JobWaiter ·············326
7.4.2 ActiveJob详解 ····························328
7.4.3 DAGSchedulerEventProcessLoop的简要介绍 ·······················328
7.4.4 DAGScheduler的组成 ················329
7.4.5 DAGScheduler提供的常用方法 ···330
7.4.6 DAGScheduler与Job的提交 ····334
7.4.7 构建Stage····································337
7.4.8 提交ResultStage ························341
7.4.9 提交还未计算的Task ·················343
7.4.10 DAGScheduler的调度流程 ······347
7.4.11 Task执行结果的处理 ··············348
7.5 调度池Pool ······································351
7.5.1 调度算法 ·······························352
7.5.2 Pool的实现 ·································354
7.5.3 调度池构建器 ·····························357
7.6 任务集合管理器TaskSetManager ···363
7.6.1 Task集合 ·····································363
7.6.2 TaskSetManager的成员属性 ······364
7.6.3 调度池与推断执行 ·····················366
7.6.4 Task本地性 ·································370
7.6.5 TaskSetManager的常用方法 ······373
7.7 运行器后端接口LauncherBackend ···383
7.7.1 BackendConnection的实现 ········384
7.7.2 LauncherBackend的实现 ···········386
7.8 调度后端接口SchedulerBackend ····389
7.8.1 SchedulerBackend的定义 ··········389
7.8.2 LocalSchedulerBackend的实现分析 ································390
7.9 任务结果获取器TaskResultGetter ···394
7.9.1 处理成功的Task ·························394
7.9.2 处理失败的Task ·························396
7.10 任务调度器TaskScheduler ··············397
7.10.1 TaskSchedulerImpl的属性 ·····397
7.10.2 TaskSchedulerImpl的初始化 ···399
7.10.3 TaskSchedulerImpl的启动 ·····399
7.10.4 TaskSchedulerImpl与Task的提交 ·······················400
7.10.5 TaskSchedulerImpl与资源分配 ···························402
7.10.6 TaskSchedulerImpl的调度流程 ······························405
7.10.7 TaskSchedulerImpl对执行结果的处理 ·····························406
7.10.8 TaskSchedulerImpl的常用方法 ···409
7.11 小结 ·······································412
第8章 计算引擎 ·····································413
8.1 计算引擎概述 ·······································414
8.2 内存管理器与执行内存 ·····················417
8.2.1 ExecutionMemoryPool详解 ·······417
8.2.2 MemoryManager模型与执行内存 ··························420
8.2.3 Uni?edMemoryManager与执行内存 ·······················421
8.3 内存管理器与Tungsten ·····················423
8.3.1 MemoryBlock详解 ·····················423
8.3.2 MemoryManager模型与Tungsten ···························425
8.3.3 Tungsten的内存分配器 ··············425
8.4 任务内存管理器 ··································431
8.4.1 TaskMemoryManager详解 ·········431
8.4.2 内存消费者 ·······················439
8.4.3 执行内存整体架构 ·····················441
8.5 Task详解 ······································443
8.5.1 任务上下文TaskContext ············443
8.5.2 Task的定义 ·································446
8.5.3 Shuf?eMapTask的实现 ··············449
8.5.4 ResultTask的实现 ·······················450
8.6 IndexShuf?eBlockResolver详解 ······451
8.7 采样与估算 ···········································455
8.7.1 SizeTracker的实现分析 ·············455
8.7.2 SizeTracker的工作原理 ·············457
8.8 特质WritablePartitionedPair- Collection ······················458
8.9 AppendOnlyMap的实现分析 ···········460
8.9.1 AppendOnlyMap的容量增长 ····461
8.9.2 AppendOnlyMap的数据更新 ····462
8.9.3 AppendOnlyMap的缓存聚合算法 ·····························464
8.9.4 AppendOnlyMap的内置排序 ····466
8.9.5 AppendOnlyMap的扩展 ············467
8.10 PartitionedPairBuffer的实现分析 ···469
8.10.1 PartitionedPairBuffer的容量增长 ······················469
8.10.2 PartitionedPairBuffer的插入 ···470
8.10.3 PartitionedPairBuffer的迭代器 ···471
8.11 外部排序器 ·········································472
8.11.1 ExternalSorter详解 ·················473
8.11.2 Shuf?eExternalSorter详解 ······487
8.12 Shuf?e管理器 ····································490
8.12.1 Shuf?eWriter详解 ··················491
8.12.2 Shuf?eBlockFetcherIterator详解 ······························502
8.12.3 BlockStoreShuf?eReader详解 ···510
8.12.4 SortShuf?eManager详解 ········513
8.13 map端与reduce端的Shuf?e组合 ······························516
8.14 小结 ·········································519
第9章 部署模式 ········································520
9.1 心跳接收器HeartbeatReceiver ·········521
9.2 Executor的实现分析 ··························527
9.2.1 Executor的心跳报告 ··················528
9.2.2 运行Task ·····································530
9.3 local部署模式 ······································535
9.4 持久化引擎PersistenceEngine ··········537
9.4.1 基于文件系统的持久化引擎 ·····539
9.4.2 基于ZooKeeper的持久化引擎 ···541
9.5 领导选举代理 ·······································542
9.6 Master详解 ···········································546
9.6.1 启动Master ·································549
9.6.2 检查Worker超时························553
9.6.3 被选举为领导时的处理 ·············554
9.6.4 一级资源调度 ·····························558
9.6.5 注册Worker·································568
9.6.6 更新Worker的最新状态············570
9.6.7 处理Worker的心跳····················570
9.6.8 注册Application··························571
9.6.9 处理Executor的申请 ·················573
9.6.10 处理Executor的状态变化 ·······573
9.6.11 Master的常用方法 ···················574
9.7 Worker详解 ································578
9.7.1 启动Worker·································581
9.7.2 向Master注册Worker ···············584
9.7.3 向Master发送心跳 ····················589
9.7.4 Worker与领导选举·····················591
9.7.5 运行Driver ··································593
9.7.6 运行Executor ······························594
9.7.7 处理Executor的状态变化 ·········599
9.8 StandaloneAppClient实现 ·················600
9.8.1 ClientEndpoint的实现分析 ········601
9.8.2 StandaloneAppClient的实现分析 ······························606
9.9 StandaloneSchedulerBackend的实现分析 ························607
9.9.1 StandaloneSchedulerBackend的属性 ····························607
9.9.2 DriverEndpoint的实现分析 ·······609
9.9.3 StandaloneSchedulerBackend的启动 ··························614
9.9.4 StandaloneSchedulerBackend的停止 ·························617
9.9.5 StandaloneSchedulerBackend与资源分配 ················618
9.10 CoarseGrainedExecutorBackend详解 ····························619
9.10.1 CoarseGrainedExecutorBackend进程 ··························620
9.10.2 CoarseGrainedExecutorBackend的功能分析 ·························622
9.11 local-cluster部署模式 ·······················625
9.11.1 启动本地集群 ····························625
9.11.2 local-cluster部署模式的启动过程 ·································627
9.11.3 local-cluster部署模式下Executor的分配过程 ·················628
9.11.4 local-cluster部署模式下的任务提交执行过程 ····························629
9.12 Standalone部署模式 ·························631
9.12.1 Standalone部署模式的启动过程 ························632
9.12.2 Standalone部署模式下Executor的分配过程 ················634
9.12.3 Standalone部署模式的资源回收 ·····························635
9.12.4 Standalone部署模式的容错机制 ······························636
9.13 其他部署方案 ·····································639
9.13.1 YARN·········································639
9.13.2 Mesos ·········································644
9.14 小结 ·······································646
第10章 Spark API ································647
10.1 基本概念·····································648
10.2 数据源DataSource ····························650
10.2.1 DataSourceRegister详解 ··········650
10.2.2 DataSource详解 ························651
10.3 检查点的实现 ···································655
10.3.1 CheckpointRDD的实现············655
10.3.2 RDDCheckpointData的实现 ····660
10.3.3 ReliableRDDCheckpointData的实现 ························662
10.4 RDD的再次分析 ·······························663
10.4.1 转换API ····································663
10.4.2 动作API ····································665
10.4.3 检查点API的实现分析 ···········667
10.4.4 迭代计算 ···································669
10.5 数据集合Dataset ·······························671
10.6 DataFrameReader详解 ·····················673
10.7 SparkSession详解 ·····························676
10.7.1 SparkSession的构建器Builder ···676
10.7.2 SparkSession的API ·················679
10.8 word count例子 ·································679
10.8.1 Job准备阶段 ·····························680
10.8.2 Job的提交与调度 ·····················685
10.9 小结 ········································689
附录 ···········································690
· · · · · · (收起)
读后感
用户评价
这本书的写作风格独树一帜，它成功地在保持极高技术严谨性的同时，营造出一种近乎文学作品般的流畅感。语言的驾驭能力极强，既能用精确的术语描述最底层的硬件交互，也能用非常形象的比喻来解释抽象的系统概念。比如，书中描述数据流通过不同处理单元时的状态转换，用的类比非常巧妙，一下子就将原本晦涩难懂的并发控制问题变得直观易懂。我特别欣赏作者在讲解性能调优时所展现出的那种“匠人精神”。他不仅告诉你“是什么”，更深入地探讨了“为什么会这样”，并提供了大量的实际案例来佐证观点。阅读过程中，我感觉自己仿佛置身于一个模拟的工程现场，亲手去调试那些棘手的Bug，去优化那些看似微不足道的毫秒级延迟。这种沉浸式的学习体验，比单纯的理论学习要有效得多，它培养的不仅仅是知识的储备，更是一种解决实际工程难题的直觉和信心。
对于任何一个有志于深入理解现代计算基础设施的人来说，这本书都是一份不可多得的财富。它不是那种读完就能立刻在简历上增加亮点的速成读物，而是一份需要投入时间和心力去啃噬的硬核经典。我特别喜欢其中对于“一致性模型”的讨论，作者没有止步于CAP理论的表面，而是深入到了Quorum协议的细节，并对比了不同一致性等级对用户体验的实际影响。这种对细节的执着和对全局视野的把握，构成了本书最大的魅力所在。它教会了我，真正的系统设计，不是堆砌时髦的技术名词，而是对资源、时间、复杂性和可靠性之间永恒博弈的艺术性掌控。读完后，我感觉自己的思维方式都被重塑了，看待任何一个软件系统，都会不自觉地从数据流、同步机制和资源分配的角度去审视，这是一种极其宝贵的、长期的思维红利。这本书，值得被反复研读和珍藏。
我通常对技术书籍的阅读体验是比较挑剔的，很多书在引入阶段总是铺垫过长，或者用一些华而不实的图表来填充篇幅，真正有价值的核心内容却寥寥无几。然而，这部作品完全没有这个问题。它的叙事节奏非常紧凑，每一页都充满了信息密度。最让我印象深刻的是作者在讲解某个复杂算法优化时，使用的那种旁征博引、层层递进的论证方式。他不像某些作者那样直接抛出结论，而是带领读者一步步回顾历史上的尝试、失败和最终的突破。这使得读者在理解最终方案的同时，也能清晰地认识到为什么其他路径走不通。这种“带着问题去解决”的叙事手法，极大地增强了阅读的代入感。我甚至不得不放慢阅读速度，时常停下来，在纸上画出流程图来辅助理解，这感觉就像是与一位经验丰富的架构师进行了一次深夜的、深入的技术对谈，那种思维的碰撞和思想的启发，是单纯通过阅读文档或API手册无法获得的宝贵体验。
坦白说，初次翻开这本书时，我有些担心内容的晦涩程度，毕竟涉及的领域跨度很大，从操作系统原理到分布式算法，无所不包。但很快，我的顾虑就烟消云散了。作者的组织结构清晰得令人赞叹，即便是在讲解最复杂的跨节点通信协议时，他也能巧妙地设置一些“休息站”，用一些小节来回顾前面已经学到的关键概念，确保读者不会在知识的海洋中迷失方向。这种结构上的精妙设计，体现了作者对读者学习曲线的深切体恤。更让我称赞的是，书中对“反模式”的探讨。他没有美化任何技术栈，而是诚实地展示了在不同约束条件下，那些看似完美的理论模型是如何在现实世界中遭遇瓶颈，以及工程师们是如何采取“务实”的妥协方案。这种坦诚和深刻的反思，让这本书的价值远远超出了技术手册的范畴，它更像是一部关于工程哲学的著作，教人如何在理想与现实之间找到平衡点。
这本书真是让人眼前一亮，读完之后感觉对整个编程范式的理解都提升了一个层次。作者没有停留在那些陈词滥调的介绍上，而是深入到那些真正能体现系统精髓的底层设计逻辑中去。尤其是关于内存管理和并行计算模型的阐述，简直是教科书级别的剖析。我记得有一章专门讲了数据结构的演化如何支撑起大规模分布式计算的效率，那种从微观到宏观的视角切换，让我这个长期在应用层打滚的开发者，第一次真切地感受到了基础架构的魅力与复杂性。书中对那些看似不言自明的设计选择背后的权衡取舍进行了极其细致的梳理，比如为什么选择某种特定的调度算法而非另一种，背后涉及到的是对延迟、吞吐量、资源隔离等多个维度的深刻理解。这种深度，远超出了市面上那些泛泛而谈的“快速入门”指南。它更像是一份精心绘制的工程蓝图，每一个组件的摆放、每一条数据流的走向，都经过了严密的数学推导和无数次的实践检验，读起来酣畅淋漓，充满了对智慧结晶的敬佩之情。
适合那些想彻底了解细节的人,照着书看,可以了解很多的细节
琐碎凌乱，全是代码片段。。。
为了讲解代码而讲解代码，原理性的知识很少有，对不起价格
琐碎凌乱，全是代码片段。。。
适合那些想彻底了解细节的人,照着书看,可以了解很多的细节
