Back to all posts

Architecting API Clients


Having used Elixir for over 2 years, I have often run into places where I have to make HTTP requests. Most of the time, I end up writing a generic way of making those requests to a website, an API client. In this post, I will be sharing my views on how to write an extensible, modular, and maintainable API client, and how to test it.

These concepts could be extended to writing any module that interacts with external dependencies

Freshbooks Classic API

I will be going over an API client for Freshbooks service. At Annkissam, we use freshbooks for invoicing and timekeeping process. Freshbooks also has an API endpoint which can be used to interact with invoices, time-entries, projects and tasks. For the purposes of this blog, let’s concentrate on only time-entries.

Looking at this guide, we can see that Freshbooks Classic uses XML and HTTP for API access, unlike the usual JSON. We can also see that the request URL is https://<subdomain>.freshbooks.com/api/2.1/xml-in, where <subdomain> is the name of the organization.

There is also a rate-limit on the number of requests in a short amount of time. We can ignore this for the purposes of this post.

All the requests made to Freshbooks Classic will be HTTP POST with XML format.

Now that we know a little about the API, let’s go ahead and start writing an API client.

This blog post is mostly focused on design and I won’t be explaining the details of the business logic or how the Freshbooks API works

Freshbooks API Client

We can start off by creating a new mix project by running: $ mix new freshbooks_api_client.

Making HTTP requests

There are many ways to make HTTP requests. We can use System.cmd to make calls to curl, we can use Erlang’s built-in http library httpc or we can choose from a variety of clients that are available in the Elixir/Erlang community. This post does a great job of comparing multiple ways of making HTTP requests and concludes that hackney is the best package.

The Elixir community has an HTTP client library named HTTPoison, which uses hackney to make HTTP requests. Therefore, it’s the most popular choice for an HTTP client in the Elixir community.

To add HTTPoison to our dependencies we can update deps in mix.exs:

def deps do
  [{:httpoison, "~> 1.0"}]
end

We can run $ mix deps.get to fetch the package.

Now our library has the ability to make HTTP requests.

Caller

Now that we have added HTTPoison, we can abstract the logic around making a request (or a call) in a Caller module. This module will define a way to make requests (be it HTTPoison, curl or any other means):

# in file lib/freshbooks_api_client/caller.ex

defmodule FreshbooksApiClient.Caller do
  @callback run(any, keyword, keyword) :: String.t()

  @doc ~S(A Simple way of accessing all Caller's features)
  defmacro __using__(_opts) do
    quote do
      import unquote(__MODULE__)

      @behaviour unquote(__MODULE__)

      def run(_, _, _) do
        raise "run/3 not implement for #{__MODULE__}"
      end

      defoverridable [run: 3]
    end
  end
end

This module defines a behaviour which a module must implement to properly make calls to Freshbooks API and work with this package. We can use this pattern to write a caller for HTTPoison and XML.

I will be using sweet_xml, a library which makes dealing with XMLs easy.

It can be added to our deps:

def deps do
  [{:httpoison, "~> 1.0"},
   {:sweet_xml, "~> 0.6"}]
end

don’t forget to run $ mix deps.get

We can use SweetXml to write an Xml helper module to convert params to XMLs.

# in file lib/freshbooks_api_client/xml.ex

defmodule FreshbooksApiClient.Xml do
  import XmlBuilder

  def to_xml(method, params) do
    # .... code not shown
    # Some logic to convert params to xml based on method
  end
end

Now we can define our HTTP + XML caller, which uses caller behaviour and Xml helper module to make requests using HTTP and XML:

# in file lib/freshbooks_api_client/caller/http_xml.ex

defmodule FreshbooksApiClient.Caller.HttpXml do
  @base_url "https://:subdomain.freshbooks.com/api/2.1/xml-in"

  use FreshbooksApiClient.Caller

  import SweetXml

  alias FreshbooksApiClient.Xml

  def run(token, subdomain, method, params, opts \\ []) do
    response = method
      |> get_request_body(params)
      |> make_request(opts, token, subdomain)

    case response do
      {:ok, %HTTPoison.Response{body: resp, status_code: 200}} ->
        status = resp |> xpath(~x"//response/@status"s)
        # https://www.freshbooks.com/developers
        # A successful call returns a status of "ok"
        # An unsuccessful call returns a status of "fail"
        {String.to_atom(status), resp}
      {:ok, %HTTPoison.Response{body: _resp, status_code: 401}} ->
        {:error, :unauthorized}
      {:ok, %HTTPoison.Error{reason: _}} -> {:error, :conn}
    end
  end

  defp get_request_body(method, params) do
    Xml.to_xml(method, params)
  end

  defp make_request(body, _opts, token, subdomain) do
    HTTPoison.post(request_url(subdomain), body, generate_headers(token))
  end

  defp generate_headers(token) do
    [auth_headers(token)]
  end

  defp auth_headers(token) do
    encoded = Base.encode64("#{token}:X")
    {"Authorization", "Basic #{encoded}"}
  end

  defp request_url(subdomain) do
    String.replace(@base_url, ":subdomain", subdomain)
  end
end

Using this library, we can make requests to Freshbooks API, as long as we have the token and subdomain.

Data Structures

In order to represent the data received from Freshbooks API, we can leverage ecto. An awesome thing about ecto is that it is a data wrapper (not just a database wrapper), which means as long as we have the data we need to represent, we can make use of Ecto.Schema and Ecto.Changeset.

We can add ecto to our dependencies by adding it to deps in mix.exs and running $ mix deps.get:

def deps do
  [{:ecto, "~> 2.1"},
   {:httpoison, "~> 1.0"},
   {:sweet_xml, "~> 0.6"}]
end

Now, let’s create a file lib/freshbooks_api_client/schema.ex and define a module FreshbooksApiClient.Schema which will be responsible for defining a schema behaviour which all our data abstractions are supposed to follow:

defmodule FreshbooksApiClient.Schema do
  @doc ~S(Freshbooks abstraction for an ecto embedded schema)
  defmacro api_schema(do: fields) do
    quote do
      @primary_key false
      embedded_schema(do: unquote(fields))
    end
  end

  @doc ~S(A Simple way of accessing all Schema's features)
  defmacro __using__(_opts) do
    quote do
      use Ecto.Schema
      import Ecto.Changeset

      import FreshbooksApiClient.Schema
    end
  end
end

By doing this we have made it easier to extend this library, as it defines ways to integrate a new resource in Freshbooks Api with this library.

Now we can use that schema behviour to create a data abstraction module for time entry (in file: lib/freshbooks_api_client/schema/time_entry.ex):

defmodule FreshbooksApiClient.Schema.TimeEntry do
  use FreshbooksApiClient.Schema

  api_schema do
    field :time_entry_id, :integer
    field :hours, :decimal
    field :date, :date
    field :notes, :string
    field :billed, :boolean

    # belongs_to :staff, FreshbooksApiClient.Schema.Staff, references: :staff_id
    # belongs_to :project, FreshbooksApiClient.Schema.Project, references: :project_id
    # belongs_to :task, FreshbooksApiClient.Schema.Task, references: :task_id
  end

  def changeset(time_entry, attrs) do
   time_entry
    |> cast(attrs, [:time_entry_id, :hours, :date, :notes, :billed, :staff_id, :project_id, :task_id])
    |> validate_required(:time_entry_id)
  end
end

Now we have defined a way to represent data that we will receive from Freshbooks API and by using ecto we can also define associations between two of the FreshbooksApiClient’s schemas in future. (shown in comments above)

Interface

We now have a data abstraction (schema) and a way to make requests (caller). We also need a way to call the correct url, with the correct request body and represent the response with the correct schema. This functionality can be a part of the schema itself, but following the Single Responsibility Principle , it should be the responsibility of a new module, an interface.

An interface is the way to do CRUD actions to an API endpoint for a given resource. It’s also responsible for translating the response received from a caller and use schema to represent it.

We can now create a generic Interface behaviour:

# in file lib/freshbooks_api_client/interface.ex

defmodule FreshbooksApiClient.Interface do
  import SweetXml

  @actions ~w(create update get delete list)a

  @typedoc ~S(Types that denote a successful response)
  @type success :: Ecto.Schema.t()

  @typedoc ~S(Types that denote an unsuccessful response)
  @type failure :: HTTPoison.Error.t() | String.t()

  @typedoc ~S(Various response types that can happen upon an interface call)
  @type response :: {:ok, success()} | {:error, failure()}

  @callback create(map, atom()) :: response()
  @callback update(map, atom()) :: response()
  @callback get(map, atom()) :: response()
  @callback delete(map, atom()) :: response()
  @callback list(map, atom()) :: response()
  @callback translate(atom(), atom(), term()) :: Ecto.Schema.t()

  @callback resource() :: String.t()
  @callback resources() :: String.t()

  @callback xml_parent_spec(atom()) :: {any, list}

  defmacro __using__(opts) do
    schema = Keyword.get(opts, :schema)
    allowed = Keyword.get(opts, :allow, @actions)
    resource = Keyword.get(opts, :resource)
    resources = Keyword.get(opts, :resources)

    quote do
      import SweetXml
      import FreshbooksApiClient.Parser

      @behaviour unquote(__MODULE__)

      def resource() do
        case unquote(resource) do
          n when is_binary(n) -> n
          nil -> raise "resource/0 not implement for #{__MODULE__}"
          _ -> raise "resource given isn't a string for #{__MODULE__}"
        end
      end

      def resources() do
        case unquote(resources) do
          n when is_binary(n) -> n
          nil -> resource() <> "s"
        end
      end

      defp schema() do
        case unquote(schema) do
          nil -> raise "schema/0 not implement for #{__MODULE__}"
          _ -> unquote(schema)
        end
      end

      def list(caller, token, subdomain, params \\ []) do
        case Enum.member?(unquote(allowed), :list) do
          true ->
            method = resource() <> ".list"
            translate(caller, :list, apply(caller, :run, [token, subdomain, method, params]))
          _ -> raise "action `:list` not allowed for #{unquote(schema)}"
        end
      end

      def get(caller, token, subdomain, params) do
        case Enum.member?(unquote(allowed), :get) do
          true ->
            method = resource() <> ".get"
            translate(caller, :get, apply(caller, :run, [token, subdomain, method, params]))
            _ -> raise "action `:get` not allowed for #{unquote(schema)}"
        end
      end

      def create(caller, token, subdomain, params) do
        case Enum.member?(unquote(allowed), :create) do
          true ->
            method = resource() <> ".create"
            translate(caller, :create, apply(caller, :run, [token, subdomain, method, params]))
          _ -> raise "action `:create` not allowed for #{unquote(schema)}"
        end
      end

      def update(caller, token, subdomain, params) do
        case Enum.member?(unquote(allowed), :update) do
          true ->
            method = resource() <> ".update"
            translate(caller, :update, apply(caller, :run, [token, subdomain, method, params]))
          _ -> raise "action `:update` not allowed for #{unquote(schema)}"
        end
      end

      def delete(caller, token ,subdomain, params) do
        case Enum.member?(unquote(allowed), :delete) do
          true ->
            method = resource() <> ".delete"
            translate(caller, :delete, apply(caller, :run, [token, subdomain, method, params]))
          _ -> raise "action `:delete` not allowed for #{unquote(schema)}"
        end
      end

      def translate(FreshbooksApiClient.Caller.HttpXml, method, {:fail, xml}) when method in [:create, :update, :get, :delete] do
        FreshbooksApiClient.Interface.translate(__MODULE__, unquote(schema), FreshbooksApiClient.Caller.HttpXml, method, {:fail, xml})
      end

      def translate(FreshbooksApiClient.Caller.HttpXml, method, {:fail, xml}) do
        raise "XML Error: #{xml}"
      end

      def translate(_, _, {:error, :unauthorized}), do: raise "Unauthorized!"

      def translate(_, _, {:error, :conn}), do: raise "HTTP Connection Error!"

      def translate(caller, method, {:ok, xml}) when method in [:create, :update, :delete, :get, :list] do
        FreshbooksApiClient.Interface.translate(__MODULE__, unquote(schema), caller, method, {:ok, xml})
      end

      def translate(caller, method, {return, _data}) do
        raise "translate/3 not implemented for #{__MODULE__} w/ (#{caller}, :#{method}, {:#{return}, data})"
      end

      defoverridable [{:resource, 0}, {:resources, 0}, {:translate, 3} | Enum.map(unquote(allowed), &{&1, 2})]
    end
  end

  def translate(_interface, _schema, FreshbooksApiClient.Caller.HttpXml, _method, {:fail, xml}) do
    parent = ~x"//response"
    spec = [
      error: ~x"./error/text()"s,
      code: ~x"./code/text()"i,
      field: ~x"./field/text()"os,
    ]

    errors = xml
    |> xpath(parent, spec)

    {:error, errors}
  end

  def translate(interface, schema, FreshbooksApiClient.Caller.HttpXml, :list, {:ok, xml}) do
    resources_key = apply(interface, :resources, [])
    per_page = xml |> xpath(~x"//response/#{resources_key}/@per_page"i)
    page = xml |> xpath(~x"//response/#{resources_key}/@page"i)
    pages = xml |> xpath(~x"//response/#{resources_key}/@pages"i)
    total = xml |> xpath(~x"//response/#{resources_key}/@total"i)

    {parent, spec} = apply(interface, :xml_parent_spec, [:list])

    resources = xml
      |> xpath(parent, spec)
      |> Enum.map(&(to_schema(schema, &1)))

    %{
      per_page: per_page,
      page: page,
      pages: pages,
      total: total,
      resources: resources,
    }
  end

  def translate(interface, schema, FreshbooksApiClient.Caller.HttpXml, :get, {:ok, xml}) do
    {parent, spec} = apply(interface, :xml_parent_spec, [:get])

    params = xml
    |> xpath(parent, spec)

    {:ok, to_schema(schema, params)}
  end

  def translate(interface, _schema, FreshbooksApiClient.Caller.HttpXml, :create, {:ok, xml}) do
    {parent, spec} = apply(interface, :xml_parent_spec, [:create])

    params = xml
    |> xpath(parent, spec)

    {:ok, params}
  end

  def translate(_interface, _schema, FreshbooksApiClient.Caller.HttpXml, :update, {:ok, _xml}) do
    {:ok, nil}
  end

  def translate(_interface, _schema, FreshbooksApiClient.Caller.HttpXml, :delete, {:ok, _xml}) do
    {:ok, nil}
  end

  def to_schema(schema, params) do
    apply(schema, :changeset, [struct(schema), params])
    |> Ecto.Changeset.apply_changes()
end

In the above module we have functions that do CRUD actions (like get, list, create etc) and functions that translate a response from a caller (like translate and to_schema). It defines a behaviour which must be followed by interfaces that correspond to a Freshbooks resource.

Let’s now use the above behaviour to write an interface for TimeEntries resource:

# in file lib/freshbooks_api_client/interface/time_entries.ex

defmodule FreshbooksApiClient.Interface.TimeEntries do
  use FreshbooksApiClient.Interface,
    schema: FreshbooksApiClient.Schema.TimeEntry,
    resources: "time_entries",
    resource: "time_entry"

  def xml_parent_spec(:list) do
    {
     ~x"//response/time_entries/time_entry"l,
      xml_spec()
    }
  end

  def xml_parent_spec(:get) do
    {
      ~x"//response/time_entry",
      xml_spec()
    }
  end

  def xml_parent_spec(:create) do
    {
      ~x"//response",
      [
        time_entry_id: ~x"./time_entry_id/text()"i,
      ]
    }
  end

  def xml_spec do
    [
      time_entry_id: ~x"./time_entry_id/text()"i,
      hours: ~x"./hours/text()"s |> transform_by(&parse_decimal/1),
      date: ~x"./date/text()"s |> transform_by(&parse_date/1),
      notes: ~x"./notes/text()"s,
      billed: ~x"./billed/text()"s |> transform_by(&parse_boolean/1),
      staff_id: ~x"./staff_id/text()"i,
      project_id: ~x"./project_id/text()"i,
      task_id: ~x"./task_id/text()"i,
    ]
  end
end

We can similarly write interfaces for other resources in future to extend the client’s functionality

Now, our api client is ready to make calls and return data represented in elixir structs:

FreshbooksApiClient.Interface.TimeEntries.list(FreshbooksApiClient.Caller.HttpXml, "api_token", "subdomain")

Writing automated tests

Now that we have a base design ready for our API client, we should write tests. In this post, I will be showing how to test only the Interfaces and Callers, as other modules are just simple enough classes without HTTP calls.

For most of the apps, we don’t want to make HTTP requests in our test environment. HTTP requests rely on external services to be up and running, and it might not always be the case while testing. Moreover, we don’t want to slow down our test suite by making a too many HTTP calls.

For the purposes of testing HTTPoison, we “trust” that it’s doing its job and that as long as we have calls delegated to HTTPoison, we shouldn’t need to test an entire HTTP request process. This can be accomplished by mocking. For this we can use libraries like meck. For the most part, if we keep the logic in the HttpXml module minimal, we would need very little mocking. The idea is to separate most of the business logic from the logic of calling, which is what we did by extracting Callers and Interfaces.

Now comes the fun part. In order to test Interfaces, we can use a mocking library too. But since we have separated and abstracted the calling logic, all we need to test an interface is a Caller which works in test environment. It doesn’t have to be an HTTP-based caller. This is where we can use the caller behaviour again to write an in-memory caller module:

# in file lib/freshbooks_api_client/caller/in_memory.ex

defmodule FreshbooksApiClient.Caller.InMemory do
  use FreshbooksApiClient.Caller

  @valid_token "valid"

  def run(token, _subdomain, method, _params, _opts \\ []) do
    case token == @valid_token do
      false -> {:error, :unauthorized}
      true -> stub_request(method)
    end
  end

  defp stub_request("time_entry.list") do
    %{
      per_page: 25,
      page: 1,
      pages: 1,
      total: 2,
      resources: [
        time_entry_1(),
        time_entry_2(),
      ],
    }
  end
  defp stub_request(method), do: %{}

  defp time_entry_1 do
    %FreshbooksApiClient.Schema.TimeEntry{
      time_entry_id: 100,
      hours: Decimal.new(8),
      date: ~D[2018-01-01],
      notes: "New Years",
      billed: false,
    }
  end

  defp time_entry_2 do
    %FreshbooksApiClient.Schema.TimeEntry{
      time_entry_id: 101,
      hours: Decimal.new(8),
      date: ~D[2018-01-01],
      notes: "New Years Day",
      billed: false,
    }
  end
end

Now we can use this module as the Caller to test Interfaces and stub calls to HTTP. This not only allowed us to write integration tests for the interface without mocking, but also to design our interface independent of the way requesting is performed. So, in future if there was, say a GraphQL endpoint for Freshbooks, we know that this interface will work as long as the caller follows the expected behaviour. By doing this we have made the interface, independent of the lower-level caller module. This is referred to as Dependency Inversion Principle.

This can further be extended by adding a configuration which takes a caller. In this way, caller can be configured to InMemory in the test environment and HttpXml in the dev and production. The information for token, subdomain and caller can also be extracted into its own data structure called api, but that’s out of the scope of this blog post. If you want to learn more about the configuration and api module, here’s the final version of freshbooks_api_client.

Final Thoughts

In this post we learned how to write an API client by separating the abstraction, calling and interface. We also saw how doing that helps us improve our package’s extensibility and write more integration tests. We also saw how using Dependency Injection to provide Caller module to Interface helps us make the code more transparent and easy to test. This approach isn’t limited to just API or HTTP. It can be applied at other places like:

I’m sure there are ways we can improve this code, but the idea was to extract each component of the app and write behaviours defining those components. There are also alternative ways of testing, like using mox which follows the principle of the famous “Mocks and explicit contracts” post, but we wanted to share another approach that followed similar principles.

We look forward to hearing your thoughts about this approach and how it compares with other ways of writing and testing API clients.


Do you love Elixir? So do we!

Annkissam has been building production applications with Elixir since 2015 and we're excited to leverage the experience we've gained over the years to help others use this amazing technology.