/ Go

gokv: Go Database Interface

A TDD-approved persistence-layer abstraction

kiwi

SQL is hard. Dealing with persistence in general is hard and error-prone. Sometimes the functionality one needs is very little over CRUD: in such cases, it may be interesting to have a library deal with the database-specific instructions.

ORMs are here for simplifying the process of wiring our code to a persistence layer, but most of the time they come with a fundamental flaw: they propose themselves as a generic implementation that will work for this and that database.

Instead of providing a generic implementation, gokv tries a different approach: it aims at defining a generic interface.

Defining a Store interface

The gokv/store.Store interface defines a basic set of high level methods, starting from the basic Set, Get, Update, Delete.

// Store defines an interface for interacting with a key-value store able to
// store JSON data in some form.
type Store interface {

	// Get retrieves a new value by key and unmarshals it to v, or returns false if
	// not found.
	// Err is non-nil if key was not found, or in case of failure.
	Get(ctx context.Context, key interface{}, v json.Unmarshaler) (ok bool, err error)

	// GetAll unmarshals to c every item in the store.
	// Err is non-nil in case of failure.
	GetAll(ctx context.Context, c Collection) error

	// Add assigns the given value to the given key if it doesn't exist already.
	// Err is non-nil if key was already present, or in case of failure.
	Add(ctx context.Context, key interface{}, v json.Marshaler) error

	// Set idempotently assigns the given value to the given key.
	// Err is non-nil in case of failure.
	Set(ctx context.Context, key interface{}, v json.Marshaler) error

	// SetWithTimeout assigns the given value to the given key, possibly
	// overwriting. The assigned key will clear after timeout. The lifespan starts
	// when this function is called.
	// Err is non-nil in case of failure.
	SetWithTimeout(ctx context.Context, key interface{}, v json.Marshaler, timeout time.Duration) error

	// SetWithDeadline assigns the given value to the given key, possibly overwriting.
	// The assigned key will clear after deadline.
	// Err is non-nil in case of failure.
	SetWithDeadline(ctx context.Context, key interface{}, v json.Marshaler, deadline time.Time) error

	// Update assigns the given value to the given key, if it exists.
	// Err is non-nil if key was not found, or in case of failure.
	Update(ctx context.Context, key interface{}, v json.Marshaler) error

	// Delete removes a key and its value from the store.
	// Err is non-nil if key was not found, or in case of failure.
	Delete(ctx context.Context, key interface{}) error

	// Ping returns a non-nil error if the Store is not healthy or if the
	// connection to the persistence is compromised.
	Ping(ctx context.Context) error

	// Close releases the resources associated with the Store.
	// Any further operation may cause panic.
	// Err is non-nil in case of failure.
	Close() error
}

Not every store implementation will, nor should, implement every defined method. A Redis store will easily implement SetWithTimeout, but it might not be a goal for it to implement FindWord (still in the works, not pictured above). The consumer code will define a subset of this interface only featuring its needed methods.

In the PostgreSQL implementation, a Store represents a database table. It is instantiated with a *sql.DB and a table name. In Redis, a Store might represent a namespaced group of entries, possibly spread across different types.

The Add behaviour is defined as: a method that persists a new key-value pair, erroring if the key is already present.

The interface tells nothing about how the implementation will comply to this.

Moreover, the interface tells nothing about the tradeoffs. The same behaviour might be implemented very efficiently in PostgreSQL, and less efficiently in Redis… or vice-versa. The implementation’s documentation is expected to provide insight. The bright side is: benchmarking the different stores with your specific use case might be a matter of changing a couple lines of code.

kiwi1

JSON as an interchange format

The gokv interface is based on JSON marshalability. In order to avoid reflection and type assertion, the methods for persisting new data receive json.Marshaler, and the methods for fetching data accept json.Unmarshaler.

The methods for fetching multiple results leverage the Collection interface, defined as:

type Collection interface {
    New() json.Unmarshaler
}

In this slice-based implementation of a Book collection, New appends a new Book to the slice and returns a pointer to it as a json.Unmarshaler:

type BookCollection []*Book

func (bc *BookCollection) New() json.Unmarshaler {
	b := new(Book)
	*bc = append(*bc, b)
	return b
}

The Store implementation, while looping through the results from the DB, will unmarshal them into the Unmarshaler returned by New. Here is how the Collection is used in the gokv/postgres implementation of GetAll:

// GetAll appends to c every item in the store.
// Err is non-nil in case of failure.
func (s Store) GetAll(ctx context.Context, c store.Collection) error {
	rows, err := s.getAllStmt.QueryContext(ctx)
	if err != nil {
		return err
	}
	defer rows.Close()

	var b []byte
	for rows.Next() {
		if err = rows.Scan(&b); err != nil {
			return err
		}
		if err = c.New().UnmarshalJSON(b); err != nil {
			return err
		}
	}
	return rows.Err()
}

Consumer code is easy to test

Test-driven developers will find another advantage to gokv: the responsibility of dealing with the actual (possibly external) persistence is totally on the Store implementation. This means that on the consumer side, we can test every single line of code without any painful mocking. An example of 100% test coverage:

// book.go

type store interface {
	Set(ctx context.Context, key interface{}, v json.Marshal) error
}

func (b Book) Save(s store) error {
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()
	
	return s.Set(ctx, b.ID, b)
}

// book_test.go

type testStore struct {
	calledWithCtx context.Context
	calledWithID  string
	calledWithV   json.Marshaler
}

func (s *testStore) Set(ctx context.Context, id interface{}, v json.Marshaler) error {
	s.calledWithCtx = ctx
	s.calledWithID = fmt.Sprint(id)
	s.calledWithV = v
	return nil
}

func TestBookSave(t *testing.T) {
	t.Run("saves a book with its ID", func(t *testing.T) {
		var (
			s    = new(testStore)
			book = Book{ID: "1-123-4", Title: "Farenheit 451"}
		)

		_ = book.Save(s)

		if deadline, ok := s.calledWithCtx.Deadline(); !ok || time.Until(deadline) > time.Second {
			t.Errorf("deadline expected to occur in less than a second, found %q", duration)
		}

		if want, have := "1-123-4", s.calledWithID; have != want {
			t.Errorf("expected book to be set with ID %q, found %q", want, have)
		}

		if want, have := book, s.calledWithV.(Book); have != want {
			t.Errorf("expected to set %+v, found %+v", want, have)
		}
	})
}

Looks cool! How can I contribute?

The project is currently little more than an experiment.

Implementations of the gokv interface exist for in-memory storage and PostgreSQL. A Redis implementation is in the works and I’m planning on a Redis-stream store. None of them is production-ready. The gokv/store interface itself lacks many important method definitions.

If you want to get involved, get in touch! Or just file a pull request. My main focus is now on defining sane interfaces for searching, but you might have another idea?

gokv: Go Database Interface
Share this