TT 向 JT 回報 Hadoop 1.x 架構 - MapReduce (MRv1) 只有一個 JobTracker (Master)，可是要管理多個 TaskTracker (Slave)！ 10 / 74 Hadoop 1.x 架構與限制 比較基本的模組： Hadoop HDFS (Storage) Hadoop MapReduce (Computing Engine + Resource JobTracker 是架構瓶頸，Concurrent Task 大概是 40,000 上下 HDFS 只能有一個 Namespace，沒辦法分開管控 /sales、/accounting、... 只能執行 MapReduce Job ... 弱弱的問一下：台灣有多少企業 Cluster 有這麼大？Task 有這麼 多？ 11 / 74 我們對 Hadoop 的期許： Batch Job Interactive (HDFS)，但是只有一種玩法 (MapReduce)？ 12 / 74 改造好呢？還是放棄好呢？ 13 / 74 進擊的 Hadoop 14 / 74 改造 MapReduce Hadoop 掌握所有資料 (HDFS)，但是提供多種玩法 (YARN)！ 希望把 Hadoop 從 Batch 應用變成 Data Operating System： 透過 MapReduce 進行 Batch Processing

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base jl:278; randCHAPTER 32. PROFILING 405 [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base jl:278; randCHAPTER 32. PROFILING 405 [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples

skipmissing(Union{Missing, Int64}[3, missing, 2, 1]) julia> maximum(x) 3 julia> sum(x) 6 julia> mapreduce(sqrt, +, x) 4.146264369941973 Objects created by calling skipmissing on an array can be indexed /reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinear,... 3 ./reduce.jl:426; mapreduce_impl(::Base.#identity, ::Base.#scalarmax, ::Array{F... 25 ./reduce.jl:428; mapreduce_impl(::Base ::Ptr... 310 ./random.jl:278; rand [....] 2893 ./REPL[1]:3; myfunc() 2893 ./reduce.jl:270; _mapreduce(::Base.#identity, ::Base.#scalarmax, ::IndexLinea... [....] In general, if you have N samples