WebHDFS >

目次

その他のバージョンのセットアップ

  1. WebHDFS1.0

セットアップ

  1. WebHDFSはHDFS組み込みですので、設定ファイル(hdfs-site.xml)を以下の要領で編集します。note.png完全分散モードではNameNodeDataNodeのサービスプリンシパルがそれぞれ異なりますが、設定キーが一つしか用意されていないようです。ノードごとに同名のシンボリックリンク(HTTP.keytab)を用意するのがベターでしょう。
    1.     <property>
    2.         <name>dfs.webhdfs.enabled</name>
    3.         <value>true</value>
    4.     </property>
    5.     <property>
    6.         <name>dfs.web.authentication.kerberos.principal</name>
    7.         <value>HTTP/localhost@${this.realm}</value>
    8.         <!-- <value>HTTP/_HOST@${this.realm}</value> -->
    9.     </property>
    10.     <property>
    11.         <name>dfs.web.authentication.kerberos.keytab</name>
    12.         <value>${this.keytab.dir}/localhost.keytab</value>
    13.         <!-- <value>${this.keytab.dir}/HTTP.keytab</value> -->
    14.     </property>

使用例

  1. リスト
    $ sudo -u alice kinit
    $ sudo -u alice bin/hadoop fs -ls /user/alice
    ...
    Found 1 items
    drwx------   - alice alice          0 2012-08-27 10:30 /user/alice/.staging
    
    $ sudo -u alice curl -i --negotiate -u : "http://localhost:50070/webhdfs/v1/user/alice?op=LISTSTATUS"
    HTTP/1.1 401 
    Content-Type: text/html; charset=iso-8859-1
    WWW-Authenticate: Negotiate
    Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
    Cache-Control: must-revalidate,no-cache,no-store
    Content-Length: 1368
    Server: Jetty(6.1.26)
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCrAv0Cz2EqzSIr7/NauwLCG3gavdYg3kuzuaS
    sqlXHJrb2WFhttsNBVXuf3xguLNTwG/JIugF6VLIH4LPYgGcBcvY
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth="u=alice&p=alice@GRID.EXAMPLE.COM&t=kerberos&e=1346070363820&s=VFTMNXgEiSQ6Csi3P/6fQiQ7qYM=";
    Path=/
    Content-Length: 221
    Server: Jetty(6.1.26)
    
    {"FileStatuses":{"FileStatus":[
    {"accessTime":0,"blockSize":0,"group":"alice","length":0,"modificationTime":1346031007360,"owner":"alice",
    "pathSuffix":".staging","permission":"700","replication":0,"type":"DIRECTORY"}
    ] } }
  2. アップロード
    $ sudo -u alice curl -i -L --negotiate -u : -X PUT -T sample.txt \
    > "http://localhost:50070/webhdfs/v1/user/alice/sample.txt?op=CREATE"
    HTTP/1.1 100 Continue
    
    HTTP/1.1 401 
    WWW-Authenticate: Negotiate
    Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
    Content-Length: 0
    Server: Jetty(6.1.26)
    
    HTTP/1.1 100 Continue
    
    HTTP/1.1 307 TEMPORARY_REDIRECT
    WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCxpWTFNSPlCLAkaQddGCcfKCHHUgiWozC6EYz
    a90rS5vm6vCkQdAL0Vz5vjRMEn9cTEJV6GEw6mqmAZkpWDVtOFyS
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth="u=alice&p=alice@GRID.EXAMPLE.COM&t=kerberos&e=1346070638510&s=8cnVquxO6ettLOeHPDBtgP5hDKA=";
    Path=/
    Location: http://deb00.grid.example.com:1006/webhdfs/v1/user/alice/sample.txt?op=CREATE&delegation=HgAFYWxpY2UFYWxpY2UAigE5Z
    eqes4oBOYn3IrMgIBT8m_XBKFHdmL6RlQziW01u_q6EwhJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA
    &namenoderpcaddress=localhost:9000&overwrite=false
    Content-Type: application/octet-stream
    Content-Length: 0
    Server: Jetty(6.1.26)
    
    HTTP/1.1 100 Continue
    
    HTTP/1.1 201 Created
    Location: webhdfs://0.0.0.0:50070/user/alice/sample.txt
    Content-Type: application/octet-stream
    Content-Length: 0
    Server: Jetty(6.1.26)
  3. 読み出し
    $ sudo -u alice curl -i -L --negotiate -u : "http://localhost:50070/webhdfs/v1/user/alice/sample.txt?op=OPEN"
    HTTP/1.1 401 
    WWW-Authenticate: Negotiate
    Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
    Content-Type: text/html; charset=iso-8859-1
    Cache-Control: must-revalidate,no-cache,no-store
    Content-Length: 1379
    Server: Jetty(6.1.26)
    
    HTTP/1.1 307 TEMPORARY_REDIRECT
    WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCNXiCHCbOYlV5BepeHiLqmEXMXZgY6vIDWKJz
    5row0n8o9LOORe+18V3uV50CdD/RESQT+H1Z4v17S7vBHzC3h//u
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth="u=alice&p=alice@GRID.EXAMPLE.COM&t=kerberos&e=1346070707652&s=DWJ9wgbaSPFFgpwTxCWu+0ractU=";
    Path=/
    Content-Type: application/octet-stream
    Location: http://deb00.grid.example.com:1006/webhdfs/v1/user/alice/sample.txt?op=OPEN&delegation=HgAFYWxpY2UFYWxpY2UAigE5Zeu
    szYoBOYn4MM0hIBRXHOa2IJ_nfWc0WrS_tQiqToKgWRJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA
    &namenoderpcaddress=localhost:9000&offset=0
    Content-Length: 0
    Server: Jetty(6.1.26)
    
    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: 15
    Server: Jetty(6.1.26)
    
    Hello WebHDFS!
  4. 削除
    $ sudo -u alice curl -i --negotiate -u : -X DELETE "http://localhost:50070/webhdfs/v1/user/alice/sample.txt?op=DELETE"
    HTTP/1.1 401 
    WWW-Authenticate: Negotiate
    Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
    Content-Length: 0
    Server: Jetty(6.1.26)
    
    HTTP/1.1 200 OK
    WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCDyM4g3MUSaIJa+NPkeWOZsxWGLUkfpdKsZQt
    1bNcRq2qTuX4WomcgPyOCvsC6CbPoqi56cdnFjyogHfrfVjCem/C
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth="u=alice&p=alice@GRID.EXAMPLE.COM&t=kerberos&e=1346070870694&s=pr4dZ0BdyTEaW9xZj5QK2lJ2jVM=";
    Path=/
    Content-Type: application/json
    Transfer-Encoding: chunked
    Server: Jetty(6.1.26)
    
    {"boolean":true}
    
    $ sudo -u alice curl -i --negotiate -u : -X DELETE "http://localhost:50070/webhdfs/v1/user/alice/sample.txt?op=DELETE"
    HTTP/1.1 401 
    WWW-Authenticate: Negotiate
    Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
    Content-Length: 0
    Server: Jetty(6.1.26)
    
    HTTP/1.1 200 OK
    WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCCQPlg+NGMyE7RV8ZYwfHDeWodIED4jUStgxT
    Or13kf9ES4URBP21iNBph0hAr7FojeDgeiM18sK1xLVDFZ6RNG53
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth="u=alice&p=alice@GRID.EXAMPLE.COM&t=kerberos&e=1346070979343&s=66HWrPcRgEvNx066yjdeA8f9PTg=";
    Path=/
    Content-Type: application/json
    Transfer-Encoding: chunked
    Server: Jetty(6.1.26)
    
    {"boolean":false}

経路の暗号化(TLS/SSL)

  • HTTPSに対応することは技術的に容易ですが、同じ通信経路を使用するHadoopネイティブプロトコルが(パフォーマンスなどの理由で)暗号化をサポートしていない以上、WebHDFSでのみ経路を暗号化してもセキュリティ上の有効性はありません。クラスタまでの機密性を確保したい場合には、まずクラスタ全体をネットワーク的に防御し、そのエッジにHttpFSなどでHDFSプロキシサーバを構築して、そこまでの経路を暗号化(HTTPS)すべきでしょう。

リソース

  1. WebHDFS REST API

トップ   編集 凍結 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2013-03-09 (土) 11:08:56 (3153d)