NoSQL database overview (incomplete, steadily growing)

• Wide Column Store / Column Families: Big Table, Hadoop, HBase, Cassandra, Hypertable, Cloudata, Cloudera
• Document Store: CouchDB, MongoDB, Terrastore, ThruDB, RavenDB, Riak, SisoDB, Amazon SimpleDB,
• Key Value / Tuple Store: Azure Table Storage, Couchbase (Membase / MemcacheDB), Redis, Chordless, GenieDB, Scalaris, Keyspace, Berkeley DB, Faircom C- Tree, HamsterDB, Pincaster, RaptorDB
• Eventually Consistent Key Value Store: Amazon Dynamo, Voldemort, Dynomite
• Graph Databases: OrientDB, Neo4J, FlockDB (twitter), Inifinite Graph, Sones, InfoGrid, HyperGraphDB, Trinity, AllegroGraph, Bigdata, DEX
• Object Databases: db4o, Versant, Objectivity, Starcounter, Perst, ZODB (Zope), NEO (Zope), PicoLisp
• Grid & Cloud Database Solutions: GigaSpaces, Hazelcast, Queplix, XML Databases, Mark Logic server, EMC Documentum xDB, Tamino, Xindice
• Multivalue Databases: U2, OpenInsight, OpenQM

see also: http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis/

Cassandra (http://cassandra.apache.org/)

• written in Java by Facebook (powers their inbox search feature)
• structured key-value store
• keys map to multiple values, stored in column families
• diff. keys can have diff. columns
• AP - eventual consistency
• Elasticity
• fault-tolerant
• Tunable consistency
• RowMutations instead of update-in-place

Usage:

• write more than read (logging)
• every component of system must be Java
• Banking, financial industry
• writes are faster than reads, real time data analysis

mongoDB (http://www.mongodb.org/)

• written in C++
• Document store (manages collections of JSON-like documents - called BSON)
• specific drivers are needed to access DB
• designed with performance in mind
• cursors for query results and can return specific fields instead of whole document
• JS expressions can be part of a query
• Replication via master/slave setups
• supports sharding

Usage:

• dynamic queries, defined indexes, no map/reduce functions
• good performance on big DB
• data changes too much for CouchDB
• RDBMS but without restrictive schema

neo4J (http://neo4j.org/)

• written in Java
• graph database, embedded or standalone
• fully transactional, supports ACID, transaction recovery, concurrency control
• disk based, schema-less
• access: REST JSON API, direct Java API
• excells on highly connected data
• focuses on a search-centric rather than compute-intensive method for data processing
• richer data model than RDBMS

Usage:

• graph-style data, Social relations
• public transport links, road maps, network topologies

riak (http://www.basho.com/Riak.html)

• written in Erlang, principles of Dynamo
• core developer team comes from Akamai
• schema-less document store
• HTTP REST and drivers for accessing data
• shard-partitioned storage
• links from one entry to other DB entries
• pluggable backend
• convenient & flexible cluster system:
  
  • configurable to be most reliable or very scalable by NRW values
  • consistent hashing is used to distribute data around the cluster

Usage:

• Factory control systems
• when even seconds of downtime hurt

Reading: http://riakhandbook.com/ by Mathias Meyer (@roidrage)

redis (http://redis.io/)

• written in C, sponsored by VMware
• key-value store, not strings only
• server side atomic operations
• whole data is held in RAM, nonblocking IO
• in-memory, persistent, journaled
• persistence is offered in two ways
• many language drivers available
• supports pipelining to reduce round trip
• replication tree via master-slave relation
• extremely fast due in memory nature

Usage:

• rapidly changing data, should fit mostly in memory
• Stock prices, Analytics, Real-time data collection, Real-time communication

try it: http://try.redis-db.com/

What is CouchDB? MVCC and ACID

MVCC - Multi Version Concurrency Control

ACID

• Atomicity - all or nothing during parallel changes
• Consitency - database is always consistent
• Isolation - no impact between write or delete operations
• Durability - changes in the DB are permanent

What is CouchDB? CAP

CAP Theorem by Dr. Eric Brewer - approved by Seth Gilbert and Nancy Lynch

• Consistency, Availability, Partition Tolerance
• only two properties can be guaranteed in distributed systems

In CouchDB when used as distributed system: AP

• C: eventual consistent - editing the same doc on two nodes will eventually be consistent (after replication)
• A: possible with master-slave replication
• P: not relevant for CouchDB when used as single node (no cluster facilities available - but projects like BigCouch, CouchDB-Lounge)

What is CouchDB? CAP

Querying CouchDB with http - db level

Hello CouchDB:

curl -X GET http://localhost:5984 
{"couchdb":"Welcome","version":"1.2.0"}
          

All DB's

curl -X GET http://localhost:5984/_all_dbs
["_replicator","_users","kassenbuch","multi-ids"]
          

Create a DB

curl -X PUT http://localhost:5984/jug
{"ok":true}
curl -X GET http://localhost:5984/_all_dbs
["_replicator","_users","jug","kassenbuch","multi-ids"]
          

Querying CouchDB with http - document level

Creating a document with POST

curl -X POST http://localhost:5984/jug/ -H "Content-Type: application/json" -d '{}'
{"ok":true,
 "id":"03cab906fa68e852be833ebb17001cd4",
 "rev":"1-967a00dff5e02add41819138abb3284d"}
          

id

rev

Annotation

Querying CouchDB with http - document level

Creating a document with PUT with usage of a uuid ...

curl -X GET http://localhost:5984/_uuids
{"uuids":["03cab906fa68e852be833ebb17000ff8"]}
curl -X PUT http://localhost:5984/jug/03cab906fa68e852be833ebb17000ff8 -d '{}'
{"ok":true,
 "id":"03cab906fa68e852be833ebb17000ff8", 
 "rev":"1-967a00dff5e02add41819138abb3284d"}
          

... or with a own id

curl -X PUT http://localhost:5984/jug/first_doc -d '{}'
{"ok":true,"id":"first_doc","rev":"1-967a00dff5e02add41819138abb3284d"}
          

Annotation

Querying CouchDB with http - document level

Updating a document with PUT

curl -X PUT http://localhost:5984/jug/first_doc 
-d '{"_rev":"1-967a00dff5e02add41819138abb3284d",
     "some-content":"Hey how are you dude?"}'
{"ok":true,"id":"first_doc","rev":"2-65ef50076c98c6817fb3ca97c87d8b66"}
          

possible to use the if-match header

-H "if-match:1-967a00dff5e02add41819138abb3284d"
          

Annotation

Querying CouchDB with http - document level

Difference between POST and PUT

• both methods are creating new documents
• the difference is the meaning of the request-uri
• "the URI in a POST request identifies the resource that will handle the enclosed entity" means: CouchDB (the resource) creates the document all by itself
• "the URI in a PUT request identifies the entity enclosed with the request" means: CouchDB creates a document with the given document id
• both methods create the document with the given request-body
• modifying a document is only possible with PUT

Annotation

Querying CouchDB with http - document level

Deleting a document with GET

curl -X DELETE http://localhost:5984/jug/first_doc?rev=3-73846195cbf86c1a3b94b6c04b6bebc0
{"ok":true,"id":"first_doc","rev":"4-9853b2dfc30eb372590c60a339124cfd"}
          

possible to use the if-match header

-H "if-match:3-73846195cbf86c1a3b94b6c04b6bebc0"
          

document is not deleted physically but marked with --deleted: true

curl -X GET http://localhost:5984/jug/first_doc
{"error":"not_found","reason":"deleted"}
          

Querying CouchDB with http - document level

Add attachment to document

curl -X PUT http://localhost:5984/jug/second_doc -d '{}'
{"ok":true,"id":"second_doc","rev":"1-967a00dff5e02add41819138abb3284d"}

curl -X PUT http://localhost:5984/jug/second_doc/mvcc.png?rev=1-967a00dff5e02add41819138abb3284d 
 --data-binary @mvcc.png 
 -H "Content-Type: image/png"
{"ok":true,"id":"second_doc","rev":"2-f488b45f8c02c4da096d23739051d428"}
          

Querying CouchDB with http - document level

Receive attachment from document

curl -i -X GET http://localhost:5984/jug/second_doc/
HTTP/1.1 200 OK
Server: CouchDB/1.2.0 (Erlang OTP/R15B01)
ETag: "2-f488b45f8c02c4da096d23739051d428"
Date: Sun, 06 May 2012 21:03:26 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 200
Cache-Control: must-revalidate

{"_id":"second_doc",
 "_rev":"2-f488b45f8c02c4da096d23739051d428",
"_attachments":
 {"mvcc.png":
  {"content_type":"image/png",
   "revpos":5,
   "digest":"md5-RJsN7gjQubg+iqLe96+SxQ==",
   "length":12753,
   "stub":true
  }
 }
}
          

Querying CouchDB with http - document level

Copy the documents content into another one

curl -i -X PUT http://localhost:5984/jug/third_doc -d '{"first_name": "Andy"}'
{"ok":true,"id":"third_doc","rev":"1-16e80e6f62d44c4569d13e2e7a05e373"}

curl -X PUT http://localhost:5984/jug/fourth_doc -d '{"last_name": "Wenk"}'
{"ok":true,"id":"fourth_doc","rev":"1-18236196ad905238b2b24d33a12eb043"}

curl -X COPY http://localhost:5984/jug/third_doc 
-H 'destination:fourth_doc?rev=1-18236196ad905238b2b24d33a12eb043'
{"ok":true,"id":"fourth_doc","rev":"2-21431c6bda57d30383987ae2510c309e"}

curl -X COPY http://localhost:5984/jug/third_doc -H 'destination:fifth_doc'
{"ok":true,"id":"fifth_doc","rev":"1-993591377f6cb544ddea933a6ed28e38"}
          

Copying third_doc to fourth_doc is overwriting not merging!

Copying third_doc to fifth_doc will create the fith_doc newly

Querying CouchDB with http - document level

Get all documents

curl -X GET http://localhost:5984/jug/_all_docs
{"total_rows":1,"offset":0,"rows":[
{"id":"03cab906fa68e852be833ebb17000ff8","key":"03cab906fa68e852be833ebb17000ff8","value":{"rev":"1-967a00dff5e02add41819138abb3284d"}}
]}
          

Retrieving data with Map/Reduce

some documents with structure

{
   "_id": "f165fc62ed5d14ca84128de1d6034a36",
   "_rev": "1-357fda3050771ca747b95c0108a21fe6",
   "type": "document_store",
   "name": "CouchDB"
}
          

simplest possible view

{
   "_id": "_design/simple",
   "_rev": "2-0404067f53ecc72bc660b8bed7576604",
   "views": {
       "all_database_types": {
           "map": "function (doc) { emit(doc.type, doc.name); }"
       }
   }
}
          

Retrieving data with Map/Reduce

firing the view

curl -X GET 'http://localhost:5984/map-reduce/_design/simple/ \
  _view/all_database_types'
{"total_rows":12,"offset":0,"rows":[
{"id":"f165fc62ed5d14ca84128de1d6036c8c","key":"column_store","value":"BigTable"},
{"id":"f165fc62ed5d14ca84128de1d60375a4","key":"column_store","value":"Hadoop"},
{"id":"f165fc62ed5d14ca84128de1d6037f0e","key":"column_store","value":"Cassandra"},
{"id":"f165fc62ed5d14ca84128de1d6038002","key":"column_store","value":"HBase"},
{"id":"f165fc62ed5d14ca84128de1d6034a36","key":"document_store","value":"CouchDB"},
{"id":"f165fc62ed5d14ca84128de1d6035544","key":"document_store","value":"MongoDB"},
{"id":"f165fc62ed5d14ca84128de1d60358c0","key":"document_store","value":"Riak"},
{"id":"f165fc62ed5d14ca84128de1d60367ac","key":"document_store","value":"Amazon SimpleDB"},
{"id":"f165fc62ed5d14ca84128de1d60387fb","key":"key_value","value":"couchbase"},
{"id":"f165fc62ed5d14ca84128de1d6038f3f","key":"key_value","value":"redis"},
{"id":"f165fc62ed5d14ca84128de1d6039f11","key":"key_value","value":"berkley DB"},
{"id":"f165fc62ed5d14ca84128de1d603a8b6","key":"key_value","value":"Raptor DB"}
]}
          

attention: start using single quotes for the URL - otherwise cURL will fuck up!

Retrieving data with Map/Reduce

firing the view and receive only data for type=column_store

curl -X GET 'http://localhost:5984/map-reduce/_design/simple/ \
  _view/all_database_types?key="document_store"'
{"total_rows":12,"offset":4,"rows":[
{"id":"f165fc62ed5d14ca84128de1d6034a36","key":"document_store","value":"CouchDB"},
{"id":"f165fc62ed5d14ca84128de1d6035544","key":"document_store","value":"MongoDB"},
{"id":"f165fc62ed5d14ca84128de1d60358c0","key":"document_store","value":"Riak"},
{"id":"f165fc62ed5d14ca84128de1d60367ac","key":"document_store","value":"Amazon SimpleDB"}
]}
          

See also: http://wiki.apache.org/couchdb/HTTP_view_API

Retrieving data with Map/Reduce

extending the view with a reduce function

{
   "_id": "_design/simple",
   "_rev": "2-0404067f53ecc72bc660b8bed7576604",
   "views": {
       "all_database_types": {
           "map": "function (doc) { emit(doc.type, doc.name); }"
           "reduce": "_count"
       }
   }
}
          

curl -X GET 'http://localhost:5984/map-reduce/_design/simple/ \
  _view/all_database_types?group=true'
{"rows":[
{"key":"column_store","value":4},
{"key":"document_store","value":4},
{"key":"key_value","value":4}
]}
          

If you want to fire the view without reduce, use the parameter reduce=false

_show functions - example with result

"shows": {
  "html": "function(doc, req){log(doc); log(req); if(doc) {return \
    '<table><tr><td>id:</td><td>' +doc._id+ \
    '</td><tr><tr><td>type:</td><td>'+doc.type+ \
    '</td></tr><tr><tr><td>name:</td><td>'+doc.name+ \
    '</td></tr><tr><td>last_update:</td><td>'+Date(doc.last_update)+ 
    '</td></tr></table>'} else {return '<h4>Für die angegebene ID' +req.id+ \
    ' gibt es kein Ergebnis</h4>'}}"
}
          

curl -X GET http://localhost:5984/map-reduce/_design/simple/_show/ \
  html/f165fc62ed5d14ca84128de1d6034a36
<table><tr><td>id:</td><td>f165fc62ed5d14ca84128de1d6034a36</td><tr><tr><td>type:</td><td>document_store</td></tr><tr><tr><td>name:</td><td>CouchDB</td></tr><tr><td>last_update:</td><td>Tue May 08 2012 21:38:03 GMT+0200 (CEST)</td></tr></table>
          

_list functions - example

  "lists": {
    "html": "function(head, req){ \
      var row, table = ''; \
      start({'headers':{'content-type':'text/html'}}); \
      table += \
        '<table width=\"300\"><tr><td><b>type:</b></td><td><b>name:</b></td></tr>'; \
      while(row=getRow()) { \
        table += '<tr><td>' +row.key+ '</td><td>' +row.value+ '</td></tr>'; \
      }; \
      send(table + '</table>'); \
    }"
  },
          

_list functions - example - results

_list functions - example - even more results

URL-Rewriting - in the _design document

       
"rewrites": [
  {
    "from": "/all",
    "to": "_list/html/all_database_types",
    "method": "GET",
    "query": {
       "reduce": "false"
    }
  },
  {
    "from": "/document_store",
    "to": "_list/html/all_database_types",
    "method": "GET",
    "query": {
       "reduce": "false",
       "key": "document_store"
    }
  },
  ...
],
          

URL-Rewriting - in the _design document

Results:

Q.E.D.